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NAVTRAEQUIPCEN  77-C-0009-1 
SUMMARY 

This  project  was  part  of  a continuing  effort  by  the  Navy  to  develop  and 
implement  improved  procedures  and  methodology  for  fleet  aviation  training. 

The  focus  of  concern  in  these  efforts  has  been  on  the  methodology  that  has 
come  to  be  labeled  as  the  Systems  Approach  to  Training  (SAT)  or  as  Instruc- 
tional Systems  Development  (ISD). 

The  present  study  examined  the  SAT/ISD  programs  for  four  specific  aircraft 
systems,  the  A-6E  (TRAM),  the  E-2C,  the  EA-6B  (ICAP),  and  the  SH-2F  (LAMPS). 

Each  of  these  efforts  covered  only  Phase  I of  the  total  SAT/ISD  process,  i.e., 
through  the  step  of  preparing  lesson  specification  documents.  Phase  II,  i.e., 
the  actual  authoring  of  lesson  materials  and  implementation  and  evaluation  of 
the  operational  training  system,  was  not  included  in  the  present  examination. 
These  four  SAT/ISD  efforts  were  conducted  for  the  Navy  by  three  different 
contractors  (one  contractor  performed  both  the  EA-6B  and  SH-2F  efforts),  and 
the  Navy  desired  that  their  different  procedures  and  methodologies  be  compared 
so  as  to  provide  a basis  for  future  improvements  in  Navy  aviation  SAT/ISD 
efforts. 

In  addition  to  examining  the  four  specific  SAT/ISD  efforts,  the  present 
study  .'Iso  examined  the  evolving  Navy  SAT/ISD  model  for  fleet  aviation  train- 
ing program  development.  That  model,  along  with  its  supporting  specifica- 
tions, instructions,  and  other  documentation,  provides  a well -organized  structure 
for  future  Navy  aviation  SAT/ISD  efforts.  The  comparisons  of  the  four  SAT/ISD 
programs  and  the  "lessons-learned"  therefrom  were  integrated  with  analysis  of 
the  model  to  suggest  future  areas  for  improving  the  model  and  its  underlying 
procedures . 

Results  of  the  comparisons  of  the  four  SAT/ISD  programs  reveal  that  the 
three  contractors  used  methodologies  that  were  quite  similar  in  an  overall 
sense,  but  that  there  were  many  areas  of  procedural  differences.  Among  such 
differences  were:  (1)  the  manner  and  extent  of  use  of  Navy  subject  matter 
expert  personnel  (SME);  (2)  the  level  of  detail  to  which  tasks  were  analyzed 
and  where  in  the  Phase  I process  the  maximum  level  of  detail  was  developed; 

(3)  the  format  and  level  of  detail  in  the  end-product  lesson  specification 
documents;  (4)  the  extent  and  clarity  with  which  SAT/ISD  procedures  were 
described;  and  (5)  the  absolute  and  relative  levels  of  effort  devoted  by  the 
contractors  to  the  various  Phase  I SAT/ISD  tasks. 
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With  reference  to  the  four  Phase  I SAT/ISD  efforts  it  is  concluded  that 
each  would  likely  provide  benefits  to  fleet  aviation  training  programs  if 
their  Phase  II  programs  are  successfully  carried  out.  Three  areas  of  short- 
coming of  these  Phase  I efforts  are  identified.  These  relate  to:  (1)  con- 
siderations pertinent  to  "in-cockpit"  psycho-motor  skills  training  (as  opposed 
to  cognitive  skills  training);  (2)  treatment  of  flight  performance  measure- 
ment; and  (3)  treatment  of  the  use  of  flight  simulators  in  training. 

With  reference  to  the  SAT/ISD  model,  seven  areas  for  possible  future 
improvement  are  suggested.  These  are:  (1)  personnel;  (2)  role  of  the  SHE; 

(3)  system  constraints;  (4)  documentation;  (5)  ISD  and  flight  skills;  (6) 
simulation;  and  (7)  measurement. 
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The  Naval  Training  Equipment  Center  has  been  responsible  for  technical 
direction  of  a series  of  efforts  aimed  at  developing  improved  fleet  aviation 
training  systems.  These  efforts  have  had  a dual  thrust:  the  development 
of  specific  training  programs,  and  improvement  and  extension  of  the  method- 
ology on  which  training  program  design  is  based.  That  methodology  has  been 
labeled  variously  as  the  Systems  Approach  to  Training  (SAT)  or  Instructional 
Systems  Development  (ISD).  The  basic  orientation  of  the  SAT/ISD  methodology 
is  that  of  careful  specification  of  job-relevant  training  objectives  in  be- 
havioral terms  and  the  systematic  design  of  training  systems  based  on  in- 
structional science  considerations. 

In  pursuit  of  its  SAT/ISD  goals,  the  Navy  has  sponsored  four  specific 
efforts  concerned  with  the  A-6E  (TRAM),  E-2C,  EA-6B  (ICAP),  and  SH-2F  (LAMPS) 
aircraft  systems,  respectively.  It  has  also  sponsored  several  efforts  aimed 
at  documenting  and  improving  the  SAT/ISD  methodology  for  future  applications. 
The  present  effort  is  part  of  the  Navy's  program  aimed  at  improving  the  SAT/ 
ISD  methodology.  In  it,  the  four  aircraft  instructional  system  efforts  are 
examined  and  compared,  and  the  Navy's  evolving  model  of  the  SAT/ISD  process 
is  analyzed.  On  the  basis  of  results  with  the  four  aircraft  programs  and 
analysis  of  the  SAT/ISD  model,  areas  are  identified  for  possible  future  im- 
provements in  the  model  and  its  application. 
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SECTION  I 
INTRODUCTION 


BACKGROUND 

This  report  sets  forth  the  results  of  a comparative  analysis  of  four 
specific  U.S.  Navy  sponsored  aviation  Systems  Approach  to  Training  (SAT) 
programs.  In  addition,  it  examines  the  general  trends  in  the  Navy's  devel- 
opment of  SAT  procedures  for  fleet  aviation  training.  The  objective  of  the 
present  effort  was  to  identify  from  those  various  activities  areas  for  im- 
provement of  the  SAT  methodology  and  its  future  applications  by  the  Navy. 

The  purposes  of  the  four  specific  SAT  programs  were  (1)  to  develop  cost 
effective  aircrew  training  programs  utilizing  the  SAT  methodology,  and  (2) 
to  evaluate  a variety  of  SAT  approaches  as  a means  of  improving  future  Naval 
Air  SAT  programs  in  terms  of  both  their  effectiveness  and  management.  The 
four  SAT  programs  cover  the  following  aircraft  systems: 

• A-6E  (TRAM) 

• E-2C 

• EA-6B  (I CAP) 

• SH-2F  (LAMPS) 

The  four  programs  were  conducted  by  contractor  agencies  working  in  con- 
junction with  Navy  personnel.  The  programs  were  under  the  technical  direction 
of  the  Naval  Training  Equipment  Center.  The  Navy  Personnel  Research  and  De- 
velopment Center  also  participated  in  the  SH-2F  program.  The  contractor 
agencies  involved  were:  (1)  Grumman  Aerospace  Corporation  (A-6E);  (2)  Cal- 
span  Corporation  (E-2C);  and  (3)  Courseware,  Inc.  (EA-6B  and  SH-2F). 

The  four  programs  each  covered  Phase  I of  a projected  two-phase  effort. 

In  this  regard,  these  aviation  SAT  programs  were  conceived,  for  manage- 
ment purposes,  as  having  a logical  break-point  in  the  series  of  steps 
between  the  activity  of  development  of  lesson  specifications  and  the 
subsequent  activity  of  authoring/production  of  course  materials.  Thus, 

Phase  I of  a SAT  effort  is  defined  for  these  programs,  generally,  as 
extending  from  job/task  analysis  through  the  development  of  lesson 
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specification  documents.-^  Phase  II  effort,  which  might  conceivably  be 
carried  out  by  an  agency  different  from  the  Phase  I executor,  would  ex- 
tend from  the  authoring/production  step  through  implementation,  evaluation, 
and  quality  control.  The  present  report  deals  only  with  the  Phase  I acti- 
vities under  the  four  programs,  though  one  (the  SH-2F)  has  progressed  on 
to  Phase  II.  The  Phase  I efforts  took  place  concurrently  during  the 
period  May  1975  - April  1976. 

The  four  programs  examined  were  initiated  in  response  to  a Navy  proposal 
request  entitled  "Systems  Approach  to  Training,"  In  the  course  of  several 
of  these  programs,  the  term  Instructional  Systems  Development  (ISD)  came 
to  be  used  in  lieu  of  the  SAT  designation.  Therefore,  in  the  present  re- 
port the  designations  SAT  and  ISD  will  be  used  interchangeably.  The  term 
ISD  has  come  into  rather  general  use  throughout  the  defense  instructional 
community  and,  thus,  is  used  here  as  a matter  of  emerging  convention.  Some 
authors  view  the  SAT  methodology  as  being  of  somewhat  wider  scope  than  ISD, 
but  this  point  is  not  of  major  concern  to  the  present  examination.  In  the 
context  of  this  report,  both  SAT  and  ISD  are  viewed  as  essentially  equiva- 
lent approaches  to  training  program  development  embodying  certain  concepts 
of  systems  analysis  and  with  defined  sequences  of  activities. 

Aircraft  Systems 

For  present  purposes,  no  great  amount  of  detail  concerning  the  four 
aircraft  involved  is  necessary.  However,  since  the  characteristics  of 
these  aircraft  systems,  i.e.,  their  general  mission  and  crew  composition, 
do  relate  to  the  mechanics  of  the  four  SAT/ISD  efforts,  a brief  description 
of  each  aircraft  is  given. 

A-6E  (TRAM).  The  A-6E  is  a two-man  subsonic  attack  aircraft.  The  crew 
consists  of  a pilot  (P)  and  a bombardier/navi gator  (B/N).  The  general 
mission  is  high  and  low  altitude  all-weather  attack.  The  TRAM  version 
(Target  Recognition,  Attack,  Multisensor)  extends  and  improves  weapons 
system  delivery  capability,  but  within  the  same  basic  mission  area. 

^Most  of  the  formal,  systematized  jxpositions  of  modern  instructional 
program  development  methodology  begin  with  some  form  of  job/task  analysis 
activity.  However,  some  recognize  and  describe  certain  critical  activities 
that  occur  prior  to  the  job/task  analysis  step.  In  the  SAT  methodology,  a 
major  activity  that  can  occur  prior  to  job/ task  analysis  is  that  of  Problem 
Analysis.  While  some  elements  of  Problem  Analysis  were  included  in  the  four 
Phase  I efforts  of  concern  here,  they  did  not  systematically  cover  that  aspect. 
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E-2C.  The  E-2C  is  a sophisticated  electronic  surveillance  aircraft  and 
carries  a crew  of  five.  The  pilot  (P)  and  co-pilot  (CP)  are  seated  forward, 
while  the  three  remaining  crewmembers— the  Combat  Information  Center  Officer 
(CICO),  the  Air  Control  Officer  (ACO),  and  the  Flight  Technician  (FT)— are 
seated  aft  in  the  Combat  Information  Center  (CIC)  compartment.  The  primary 
mission  of  the  E-2C  is  to  provide  Airborne  Early  Warning  (AEW)  information 
in  defense  of  a fleet  force.  In  addition,  it  can  perform  in  a variety  of 
other  missions  such  as  Surface-Subsurface-Surveillance  Control  (SSSC),  Anti-  ; 

Submarine  Warfare  (ASW)  Support,  Search  and  Rescue  (SAR),  and  others.  ! 

EA-6B  (ICAP).  The  EA-6B  is  a tactical  jaitming  aircraft  that  employs  a ] 

variety  of  Electronic  Countermeasures  (ECM).  The  four-man  crew  consists  of  ! 

a pilot  (P)  and  co-pilot  (CP),  who  occupy  the  forward  cockpit,  and  two  elec- 
tronic countermeasure  operators  (ECMO),  who  occupy  the  aft  cockpit.  The 
basic  mission  is  to  support  strike  aircraft  and  ground  forces  by  suppressing 
or  jamming  enemy  electronic  activity.  The  ICAP  version  (Improved  Capability) 
involves  latest  state-of-the-art  electronic  equipment. 

SH-2F  (LAMPS).  The  SH-2F  is  a helicopter,  used  primarily  in  Anti- 
Submarine  Warfare  (ASW)  and  in  surface  surveillance.  It  can  also  deliver 
submarine-homing  torpedoes.  There  are  three  aircrew  positions,  pilot  (P), 
co-pilot  (CP),  and  sensor  operator  (SENSO).  In  addition  to  ASW,  the  SH-2F 
is  used  for  other  missions  such  as  SAR  and  external  cargo  transport. 

Documentation 

The  Phase  I SAT  efforts  for  these  four  aircraft  systems  have  produced 
a variety  of  documentation  including  task  listings,  behavioral  objectives, 
objectives  hierarchies,  training  support  requirements,  media  selections, 
lesson  specifications,  and  a summary  technical  report.  These  documents  are 
quite  voluminous  and  vary  soirrewhat  in  format  from  one  program  to  another. 

While  most  of  the  documents  produced  were  available  for  the  present  analysis, 
some  were  not.  A listing  of  the  documents  examined  is  contained  in  the 
references  section  of  this  report. 

In  addition  to  the  documentation  of  the  four  aircraft  SAT/ISD  programs 
mentioned,  a variety  of  other  documents  pertinent  to  ISD  methodology  and 
applications  was  examined  in  the  present  review.  These  documents  are  also 
identified  in  the  references  section.  The  most  important  of  these,  for 
present  purposes,  are  a series  of  documents  dealing  with  development  and  | 
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implementation  of  a detailed  ISD  model,  including  a draft  Specification  for 
Instructional  Systems  Development.  These  items  are  being  developed  by  Course- 
ware, Inc.  and  Logicon  for  the  Navy.  This  specification  and  related  documents 
seek  to  codify  an  ISD  methodology  tailored  specifically  to  the  requirements 
of  Naval  Aviation  operator  and  maintenance  training. 

INSTRUCTIONAL  SYSTEMS  DEVELOPMENT 

The  approach  that  has  come  to  be  labeled  as  Instructional  Systems 
Development,  or  ISD,  has  evolved  over  the  last  25  years  or  so  from  the  efforts 
of  many  R & D groups,  largely  working  in  a military  training  context.  It 
is  based  on  use  of  the  so-called  systems  approach  in  the  solution  of  training 
problems.  The  ISD  and  SAT  methodologies  have  been  documented  by  many  authors 
and  will  not  be  reviewed  here.  However,  while  many  applications  of  SAT  and 
ISD--generally  by  R & D groups— have  produced  impressive  results,  there  seems 
to  be  a strong  feeling  on  the  part  of  some  that  many  such  applications  have 
failed  to  live  up  to  the  expectations  that  have  been  developed  with  reference 
to  cost  effectiveness  of  ISD.  For  example,  in  the  RFP^  for  the  recent  USAF 
procurement  of  an  F-I6  Pilot  Training  System  Program,  it  is  stated  (State- 
ment of  Work,  p.  2-4) : 

"1.2  Previous  Air  Force  applications  of  ISD,  while  yielding  worthwhile 

results,  in  many  cases  have  fallen  short  of  total  effectiveness  (p.  2) 


"2.3  Instructional  media  selection  has  been  made  intuitively  in  the 
past.  . . Compilation  of  cost  effectiveness  data  for  the  variety  of 
media  options  available  to  the  instructional  system  designer,  tradition 
ally  a conspicuous  unknown  in  prior  ISD  efforts,  will  also  be  a pri- 
mary component  of  the  F-16  training  system  procurement,  (p.3) 

"2.4  A major  criticism  that  can  be  made  of  some  previous  ISD  efforts 
is  that  these  efforts  fail  to  employ  fully  recent  advances  in  training 
technology.  Indeed,  ISD  efforts  have  been  content  to  apply  traditional 
aircrew  training  techniques  rather  than  venture  into  new  innovative 
approaches  to  training,  (p.  3-4) 

"2.5  A final  deficiency  in  previous  ISD  efforts  which  this  contract 
seeks  to  avoid  is  the  lack  of  total  training  system  optimization." 

(p.  4) 


--  USAF  RFP  No.  F02604-77-09010:  F-16  Pilot  Training  System  Program. 

Procurement  Division  (L6PK),  Luke  AFBTKTi  1 December  1976. 
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Other  sources  have  raised  similar  questions  about  ISO's  effectiveness 
in  practice.  For  example,  Montemerlo,-'^  building  on  remarks  by  Cream,  ^ 
cites  various  "unsupportable  assumptions"  as  underlying  the  sometimes  failure 
of  ISO  to  fulfill  the  benefits  expectations  of  training  and  R & D managers. 

Whether  one  accepts  the  contention  that  ISO  problems  stem  from  "unsup- 
portable assumptions,"  which  would  suggest  a conceptually  flawed  methodology, 
or  from  other  factors,  there  is  general  agreement  that  problems  can  result 

during  a specific  application  of  ISO  or  SAT  unless  there  is  a proper  blend 

5 / 

of  personnel  skills  and  resources.  In  the  final  report-  (draft)  of  the 
SH-2F  Phase  I effort,  the  author  states  one  aspect  of  the  personnel  problem 
somewhat  colorfully:  "The  Systems  Approach  to  Training  requires  the  marriage 
of  academic  rigor  and  practicality  along  with  some  of  the  characteristics 
of  Waldo  Pepper  and  P.T.  Barnum." 

The  problems  with  ISO  are  described  in  somewhat  more  prosaic  language 

in  another  of  the  ISO  Phase  I reports 

"THE  ISD  PROBLEM.  At  the  time  the  EA-6B  ISO  project  was  being  con- 
ceived, a number  of  serious  questions  about  ISD  were  in  the  minds  of 
those  concerned  with  Naval  Air  Training. 

"At  the  most  general  level,  the  problem  was  whether  or  not  the  Navy 
should  continue  to  go  with  ISD  as  the  required  approach  to  training.  . . 
The  general  feeling  was  that  ISD  provided  a framework  that  should  lead 
to  the  development  of  instructional  programs  that  are  effective  and 
efficient  in  developing  job  skills.  In  spite  of  this,  it  was  very  hard 
to  present  convincing  evidence  that  ISD  is  workable,  effective,  or 
efficient  in  terms  of  resource  consumption.  There  were  two  reasons 
for  this  lack  of  certainty. 


■s’? 

--  Montemerlo,  M.D.  "Instructional  System  Development:  Implications 
for  Further  Research,"  Proceedings  of  the  5th  Symposium  on  Psychology  in 
the  Air  Force.  U.S.  Air  Force  Academy,  CO,  April  1976,  pp.  I2l-124 

4 / 

-'Cream,  B.W.  Air  Force  ISD  Conference.  The  Pentagon,  Washington,  DC, 
3-5  February  1976. 

-^Gibbons,  A.S.  SH-2F  (LAMPS)  Instructional  Design  and  Development: 
Final  Technical  Report.  Phase  I.  Draft  Final  Report.  Courseware,  Inc., 

San  Diego,  CA,  May  1976. 

-^Hughes,  J.A.,  and  Hymes,  J.P.  A Study  of  the  Effectiveness,  Feasi- 
bility,  and  Resource  Req^uirements  of  Instructional  Systems  Development: 
EA-6B  Readiness  Training.  TecFnica!  Report  NAVTRAEQUIPCEN  75-C-OlDO-l. 
Naval  Training  Equipment  Center,  Orlando,  FL,  January  1977. 
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"First,  almost  all  versions  of  ISO  were  seriously  lacking  in  detailed, 
prescriptive  guidance  for  conducting  the  various  analysis  and  design 
phases  which  ISO  requires.  . . In  other  words,  ISO  approaches  tended 
to  describe  broad  steps  or  phases,  but  did  not  incorporate  the  detailed 
models,  algorithms,  and  techniques  from  instructional  science  and 
technology  which  would  make  the  approach  really  work  in  a standardized 
way  in  a typical  training  environment. 

"Second,  many  ISD  applications  did  not  make  adequate  provisions  for 
necessary  resources:  personnel,  time,  facilities,  or  money. 

"Given  these  two  limiting  factors,  it  is  no  wonder  that  some  casual 
observers  were  ready  to  conclude  that  ISD  was  a good  idea  that  didn't 
work."  (pp.  14-15) 

This  quotation  highlights  some  of  the  reasons  for  Navy  concern  in  these 
four  ISD/SAT  efforts  with  the  process  or  methodology  of  ISD  and  the  extent  to 
which  it  can  be  used  in  a prescriptive  manner  in  future  programs  in  Naval 
Aviation.  Thus,  while  all  four  efforts  sought  to  produce  (assuming  subse- 
quent execution  of  Phase  II)  effective  training  systems,  each  sought  to 
examine  the  procedures  used  and  to  suggest  ways  of  improving  them. 

It  is  clear,  however,  that  despite  such  problems  ISD  and  related  approaches 
do  have  much  to  offer  in  the  way  of  more  effective  training.  While  one  may 
argue  with  various  specifics  of  the  method,  few  seriously  question  the  strength 
and  advisability  of  the  general  underlying  concepts  that  are  common  to  the 
various  systems  or  applications  of  SAT  or  ISD.  One  measure  of  the  validity 
of  this  assertion  is  the  magnitude  of  ISD  activity  that  abounds  in  military 
training  circles,  as  well  as  that  in  non-military  training  and  education. 

The  emphasis  in  ISD  on  defining  desired  outcomes  in  specific  behavioral  terms, 
on  developing  the  most  cost  effective  means  of  achieving  those  outcomes,  and 
on  objective  program  evaluation  and  revision  are  its  strength  and  imbue  it 
with  an  aura  of  validity,  acceptability,  and  common  sense. 

The  growth  of  ISD  activity,  however,  presents  another  set  of  problems 
of  concern  to  the  Navy  in  its  role  as  a buyer  of  services  and  training 
products.  If  ISD  programs  are  going  to  be  procured  from  outside  vendors,  as 
in  the  case  of  the  four  aircraft  ISD  efforts  discussed  here,  not  only  must 
they  be  based  on  a methodology  that  is  conceptually  sound,  but  the  methodology 
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must  be  one  that  Is  feasible  of  management--^  by  the  procuring  agency.  Thus, 
there  is  a concern  that  the  ISO  procurement  contracts  adequately  describe 
what  is  to  be  delivered  in  a manner  that  protects  the  government's  interests 
and  maximizes  the  likelihood  that  the  ISO  effort  will  be  successful  in  pro- 
ducing an  effective  training  program.  It  is  toward  this  end  that  Navy  efforts 
to  articulate  an  effective  ISO  model,  in  both  the  technical  and  management 
senses,  have  been  directed. 

ANALYSIS  OF  NAVY  ISO  PROGRAMS 

As  noted,  a major  concern  of  the  present  report  is  a comparative  analysis 
of  the  four  Navy  aviation  ISO  programs.  The  analysis  is  based  on  the  avail- 
able program  documentation  and  the  author's  experience  with  similar  programs. 
The  basic  intent  is  to  provide  observations  and  comments  that  will  enable 
future  Navy  aviation  ISO  programs  to  be  more  effective  and  efficient  and  that 
will  provide  aid  in  the  management  of  such  programs. 

Approach 

The  approach  used  is  analytical  examination  of  the  four  aviation  ISO 
programs  as  represented  in  their  various  documentations.  In  addition,  the 
proposed  model  and  specifications  for  future  ISO  efforts  are  examined. 

Aspects  of  these  various  programs  and  the  documentation  are  interpreted, 
compared,  evaluated,  and  related  to  the  views  and  experiences  of  the  author 
and  that  of  other  researchers  and  practitioners,  as  appropriate. 


•^The  problems  of  managing  an  ISO  program  procured  from  a civilian 
contractor  are  not  fundamentally  different  from  those  that  would  be  involved 
in  management  of  an  in-house  ISO  effort  by  a military, or  civil  service  group. 
The  desired  end  product  (an  effective  training  program)  is  identical,  and 
the  production  process  is  much  the  same.  However,  for  a contracted  effort 
in-house  command  authority  functions  must  be  cast  in  contractual/legal  form, 
specifications  must  be  used,  and  various  interim  and  final  products  must  be 
defined  in  contractual /legal  terms  to  insure  that  the  process  is  executed 
as  desired.  It  is  a matter  of  which  authority  structure  is  used,  command 
or  legal.  Functions  are  much  the  same. 
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Among  the  areas  of  concern  in  this  analysis  are  the  following: 

1.  Comparative  analysis  of  the  four  SAT/ISD  programs. 

2.  Utility  of  Phase  I products  for  Phase  II. 

3.  Utility  of  products  for  Navy  decisions. 

4.  Adequacy  of  RFP  and  ISO  specifications. 

5.  Changes  to  ISO  methodology. 

The  bulk  of  the  activity  relates  to  the  first  of  the  above  areas,  which, 
in  turn,  provides  input  to  the  other  four  areas.  Where  possible,  com- 
parisons are  made  in  objective  or  quantitative  terms,  e.g.,  number  of  hours 
devoted  to  a given  activity.  In  many  instances,  however,  comparisons  and 
evaluations  necessarily  involve  a degree  of  subjectivity. 

In  comparing  the  four  SAT/ISD  programs,  the  areas  shown  in  Table  1 are 
examined.  In  some  cases  the  information  available  is  not  complete.  It 
must  be  recognized  also  that,  while  there  are  many  common  features  or 
aspects  of  the  various  programs  and  the  environment  in  which  they  were 
conducted,  there  are  many  unique  or  unusual  aspects  of  each.  Further,  while 
the  manner  in  which  a given  function  or  requirement  was  handled  in  one  pro- 
gram might  appear  more  or  less  advantageous  than  its  handling  in  another 
program,  there  may  have  been  quite  adequate  reasons  for  the  differences.  It 
is  likely  that  some  of  these  reasons  are  obscured  or  not  present  in  the 
available  documentation. 

TABLE  1.  DIMENSIONS  OF  ISO  PROGRAM  COMPARISON 

1.  Composition  of  the  ISD  Team 

A.  Number  and  types  of  personnel 

(1)  Contractor 

(2)  Navy 

B.  Qualifications 

C.  Training 

D.  Functional  roles 

E.  Navy  SME  role 

2.  Task  Listing 

A.  Information  sources 

B.  Number  of  positions/tasks 

C.  Task  validation  procedure 
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TABLE  1.  DIMENSIONS  OF  ISD  PROGRAM  COMPARISON  (CONT'D) 

D.  Amount  of  effort 

3.  Task  Selection 

A.  Method 

B.  Input  trainee  level  determination 

C.  Amount  of  effort 

4.  Task  Analysis  (Objectives  Hierarchy) 

A.  Method 

B.  Learning  orientation 

C.  Format  and  level  of  detail 

D.  Anrount  of  effort 

5.  Media  Selection 

A.  Method 

B.  Media  types 

C.  Consideration  of  alternatives 

D.  Amount  of  effort 

6.  Course  Design  (Sequence  and  Structure) 

A.  Method 

B.  Use  of  simulation 

C.  Strategies 

D.  Amount  of  effort 

7.  Lesson  Specifications 

A.  Method 

B.  Format 

C.  Flight  vs.  non-flight 

D.  Completeness 

E.  Amount  of  effort 

8.  Measurement  and  Evaluation 

A.  Flight  vs.  non-flight 

B.  Internal  evaluation 

C.  External  evaluation 

D.  Amount  of  effort 

9.  Documentation 

A.  Nature  of  reports/documentation 
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B.  Clarity 

C.  Specificity  of  procedures,  algorithms,  etc. 

D.  Amount  of  effort 
10.  General  Factors 

A.  General  program  description 

B.  Program  cost 

C.  Program  quality  (adequacy  of  methodology  and  execution) 

D.  Manageability  (from  Navy  perspective) 

E.  Implications  for  future 

Organization  of  Report 

The  report  is  organized  Into  three  general  parts:  Section  I - Introduc- 
tion; Section  II  - Resul ts ; and  Section  III  - Commentary.  The  structure  of 
the  Results  section  will  follow  somewhat  that  implied  in  Table  1 and  the  pre- 
ceding discussion,  but  it  will  depart  from  this  as  necessary  for  exploration 
or  exposition  of  various  points  of  interest.  Due  to  the  general  characteris- 
tics of  analytic  studies  of  the  present  type--i.e.,  their  lack  of  a rigorous 
scientific  evaluative  structure,  the  incompleteness  of  available  information, 
the  variety  of  circumstances  surrounding  the  individual  efforts,  the  subjec- 
tive and  interpretive  nature  of  the  evaluative  statements,  and  similar  factors 
the  final  section  is  labeled  as  Commentary  rather  than  the  more  usual  Conclu- 
sions and  Recommendations.  As  will  be  seen,  certain  conclusions  and  recom- 
mendations are  drawn  or  implied,  but  it  is  recognized  that  other  alternatives 
may  be  equally  tenable.  Therefore,  any  conclusions,  recommendations,  or 
discussions  are  more  properly  labeled  as  commentary.  The  hope  is,  of  course, 
that  the  comments  will  materially  assist  the  Navy  in  the  future  conception, 
procurement,  execution,  and  management  of  aviation  ISO  programs. 
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SECTION  II 
RESULTS 

OVERVIEW  OF  PROGRAMS 

As  noted  in  the  preceding  discussion,  results  concerning  the  four  SAT/ISD 
programs  will  be  presented  somewhat  in  accord  with  Table  1,  but  with  digres- 
sion, as  required.  Before  getting  into  dimensional  comparisons,  a brief 
overview  description  of  each  of  the  four  programs  will  be  presented. 

A-6E  (TRAM) 

The  ISO  approach  followed  by  Grumman  in  the  A-6E  effort  was  quite  similar 
to  that  described  in  the  USAF  instructional  design  handbook.-^  Seven  major 
steps  were  identified  for  the  Phase  I effort:  (1)  determine  job  performance 
requirements  and  standards;  (2)  determine  training  requirements;  (3)  determine 
terminal  and  supporting  objectives;  (4)  develop  criterion  performance  tests; 

(5)  perform  media  analysis;  (6)  plan  instructional  materials/methods;  and  (7) 
prepare  lesson  specifications.  The  contractor  team  was  made  up  of  training 
psychologists,  educational  specialists,  and  flight  test  personnel.  Navy  SME 
(subject  matter  expert)  personnel  participated  principally  in  a review  role. 
Approximately  700  tasks  were  identified  for  the  two-man  crew  (pilot  and  bom- 
bardier/navigator), from  which  373  specific  behavioral  objectives  (SBO)  were 
derived  for  training.  Some  55  different  lesson  specifications  were  developed. 
Total  ISO  Phase  I contract  cost  was  $195,000. 

E-2C 

The  approach  used  by  Calspan  in  the  E-2C  program,  which  is  also  based  on 
USAF  procedure,  is  an  expansion  of  the  methodology  used  by  the  contractor  in 
a prior  ISO  effort  on  the  B-1  stratetic  bomber  system.  Seven  major  Phase  I 
activities  are  specified  in  this  approach:  (1)  perform  task  analysis;  (2) 
select  tasks  for  training;  (3)  determine  course  objectives;  (4)  select  in- 
structional methods  and  media;  (5)  determine  instructional  strategies;  (6) 

. sequence  instruction;  and  (7)  organize  objectives  into  unit  and  lesson  structure. 

w 

- U.S.  Air  Force.  Handbook  for  Designers  of  Instructional  Systems:  Vols. 
I-V.  AFP  50-58.  Headquarters,  U.S.  Air  Force.  Washington,  DC,  July  1973. 
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The  contractor  team  composition  was  similar  to  that  described  for  the  A-6E. 

Navy  SMEs  were  used  in  minimal  fashion  in  the  program.  The  number  of  crew- 
member tasks  identified  could  not  be  specifically  determined  from  the  docu- 
mentation available.  However,  some  159  pilot/co-pilot  behavioral  objectives 
are  identified,  as  are  some  466  behavioral  objectives  for  the  other  three 
crew  positions  (CICO,  ACO,  and  FT).  The  P/CP  course  has  19  modules  which 
contain  a total  of  92  lessons.  The  NFO/FT^^  course  has  38  modules  with  a 
total  of  137  lessons.  Total  Phase  I contract  cost  was  $270,194. 

EA-6B  (I CAP) 

While  conceptually  much  the  same  as  the  two  previous  programs,  the  ISO 
model  used  by  the  EA-6B  contractor.  Courseware.  Inc.,  was  based  on  the  U.S. 
Army's  systems  engineering  of  training  model.— The  steps  involved  are:  (1) 
task  listing;  (2)  job  analysis  survey;  (3)  selection  of  tasks  for  training; 

(4)  development  of  objectives  hierarchies;  (5)  course  sequencing;  (6)  media 
selection;  and  (7)  development  of  lesson  specifications.  The  contractor  team 
consisted  of  an  instructional  psychologist,  an  instructional  technologist,  and 
a pilot  SME.  Additional  professional  support  was  provided  by  other  contractor 
personnel.  Considerable  assistance  was  provided  by  Navy  SME  personnel  in 
virtually  all  activities  in  the  program.  While  difficult  to  determine  pre- 
cisely from  the  documentation,  approximately  245  tasks  were  identified  for 
the  pilot,  co-pilot,  and  two  ECMO  crewmembers.  Additional  elements  were 
identified  for  many  of  these  tasks.  The  pi lot/ co-pi  lot  course  is  divided  into 
14  units  comprised  of  129  lessons,  while  the  ECMO  course  is  comprised  of  17 
units.  The  number  of  lessons  in  one  ECMO  unit  could  not  be  determined,  but 
there  were  129  lessons  in  the  remaining  16  units.  Cost  of  the  Phase  I contract 
was  $216,679.  However,  a substantial  portion  of  the  lesson  specifications 
was  not  completed  within  the  original  effort. 

-^The  CICO  and  ACO  are  Naval  Flight  Officers  (NFO),  whereas  the  FT  is  an 
enlisted  crewmember. 

^-'^U.S.  Army  Training  and  Doctrine  Command.  Systems  Engineering  of  Training. 
TRADOC  Regulation  350-100-1.  Fort  Monroe,  VA,  February  19o8. 
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SH-2F  (LAMPS) 

The  ISO  approach  followed  for  the  SH-2F  was  the  same  as  that  described 
above  for  the  EA-6B.  The  same  contractor.  Courseware,  conducted  both  pro- 
grams. The  contractor  ISO  team  consisted  of  an  instructional  psychologist 
and  two  instructional  development  technicians.  A substantial  number  of  Navy 
SMEs  were  involved,  as  well  as  a technical  monitor  from  the  Navy  Personnel 
Research  and  Development  Center.  Approximately  183  pi  lot/ co-pi lot  and  31 
aircrewman  (SENSO)  major  tasks  were  identified.  Additional  subtasks  and 
elements  were  identified  below  the  major  task  level,  particularly  for  the 
SENSO.  The  pilot/co-pilot  course  is  divided  into  seven  sections  comprised 
of  25  units  and  64  lessons.  The  SENSO  course  has  16  sections,  38  units,  and 
85  lessons.  Total  cost  of  the  Phase  I contract  effort  could  not  be  determined. 

COMPOSITION  OF  ISO  TEAMS 

All  four  of  the  programs  involved  some  coniji nation  of  contractor  effort 
with  that  of  the  Navy  SMEs.  Three  of  the  contractor  ISO  teams  apparently 
contained  one  or  more  persons  who  were  SMEs  in  the  flying  content  area.  From 
the  information  available,  no  contractor  SME  could  be  identified  on  the 
Courseware  SH-2F  effort.  One  contractor  (Cal span)  reported  having  its  SME 
attend  Navy  E-2C  classes  for  a two  month  period  to  increase  his  subject 
matter  expertise.  The  contractors  apparently  felt  (with  the  possible  exception 
of  the  SH-2F  program)  a requirement  for  SME  in-house  representation  in  addi- 
tion to  whatever  input  Navy  SMEs  provided. 

Additional  contractor  team  members  are  described  variously  as:  (1) 
instructional  psychologist  and  educational  specialists  (Grumnan,  A-6E);  (2) 
specialists  in  the  disciplines  of  psychology,  education  and  human  factors 
engineering  (Calspan,  E-2C);  (3)  instructional  psychologist,  instructional 
technologist,  technical  monitor,  consulting  and  training  support  group,  and 
management  monitor  (Courseware,  EA-6B);  and  (4)  instructional  psychologist 
and  instructional  design  technician  (Courseware,  SH-2F). 

Little  can  be  stated  about  the  detailed  qualifications  and  experience 
of  the  contractor  ISO  team  personnel.  All  four  contractors  generally  make 
some  claim  to  having  represented  an  instructional  psychologist,  an  instruc- 
tional technologist/technician,  and  an  aviation  SME.  While  this  seeming 
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congruity  of  staffing  might  be  interpreted  as  reflecting  a consensus  as  to 
personnel  requirements  for  ISD,  it  must  be  noted  that  the  staffing  pattern 
was  generally  dictated  by  the  RFP  requirement  that  "...  the  contractor 
snail  provide  instructional  psychologists,  education  specialists,  systems 
analysts,  and  former  military  pilots/NFO  with  relevant  experience." 

Navy  SME  Role 

Beyond  this  commonality  of  staffing  and  the  fact  that  Navy  SMEs  played 
a part  in  all  four  programs,  there  were  some  wide  variations  in  level  of 
effort  and  functional  roles  of  the  various  personnel.  In  particular,  the 
role  of  Navy  SMEs  in  the  ISD  efforts  varied  considerably.  Likely,  this 
variation  reflects  both  the  organizational  and  philosophical  orientation  of 
the  contractor,  as  well  as  situational  factors  relating  to  SME  availability, 
SME  turnover,  and  squadron  attitudes. 

The  level  of  Navy  SME  effort  reported  fcr  each  of  the  programs  is  shown 
in  Table  2.  As  can  be  seen,  the  level  of  detail  with  which  that  effort  can 
be  allocated  to  specific  ISD  tasks  varies  from  none  to  considerable.  There- 
fore, lack  of  a specific  entry  in  Table  2 for  a given  ISD  task  does  not  mean 
necessarily  that  no  Navy  SME  time  was  devoted  to  that  task  in  a particular 
program.  For  example,  considerable  SME  time  (1,348  hours)  was  devoted  to 
the  A-6E  program,  but  no  information  was  available  as  to  its  distribution 
over  the  ISD  tasks.  Also,  it  is  likely  that  the  extent  to  which  these  data 
reflect  actual  Navy  SME  effort  is  quite  variable.  The  report  of  the  E-2C 
program  (Cal span)  shows  a total  of  only  87  man-hours  of  Navy  SME  time,  but 
notes  that  this  covers  only  direct  consultation  and  does  not  involve  time 
devoted  to  "independent  review  of  supporting  documentation  (duration  not 
determinable)."  Only  in  one  program  (EA-6B)  does  the  contractor  (Courseware) 
discuss  a specific  procedure  or  mechanism  for  keeping  track  of  distribution 
of  effort  over  the  various  ISD  tasks.  By  far,  the  greatest  actual  use  of 
Navy  SME  effort  was  that  shown  in  Courseware's  SH-2F  program. 
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TABLE  2.  NAVY  SME  EFFORT®  BY  TASK  AND  PROGRAM 


Program 

ISO  Task 

A-6E 

E-2C  EA-6B 

sjir?! 

1. 

Task  Analysis 

40 

a.  Job  Analysis 

247 

640 

b.  Task  Selection 

OJ 

4-> 

80 

2. 

Training  Objectives 

o 

Q. 

19 

a.  Hierarchy  Analysis 

Vi/ 

S- 

208 

1,392 

b.  Sequencing  and  Grouping 

o 

c 

97 

160 

3. 

Media  Selection 

in 

4. 

Instructional  Design 

4-> 

O 

a.  Instructional  Objectives 

28 

b.  Lesson  Specifications 

995 

3,480 

TOTALS 

1,348 

87  1,547 

5,752 

®Time  in  man-hours 
TASK  LISTING 

The  usual  starting  point  for  an  ISO  effort  involves  defining  the  uni- 
verse of  job  behaviors  to  which  the  training  system  must  respond.  For  this 
reason,  a job/task  analysis  is  performed  in  which  all  tasks  required  in 
the  operational  job  situation  are  delineated  and  analyzed.  The  procedures 
used,  the  format,  and  the  level  of  detail  may  vary,  but  the  function  is 
common  to  all  comprehensive  ISO  efforts.  The  initial  aspect  of  the  task 
analysis  is  a listing  of  all  job  tasks  that  occur  on  the  job. 

Sources 

There  was  considerable  commonality  in  the  manner  in  which  operator 
task  listings  were  developed  in  the  four  programs.  Generally,  various 
documentary  sources  were  consulted  (e.g.,  engineering  documents,  NATOPS, 
etc.)  and  SMEs  were  utilized  to  expand  task  lists  by  "walking  through" 
typical  missions.  SME  input  was  quite  important  in  this  aspect  of  the 
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ISD  programs.  The  results  of  the  task  listing  were  usually  verified,  at  least 
to  some  degree,  by  Navy  SMEs  who  had  not  participated  in  their  initial  devel- 
opment. 

Variations  of  note  included:  (1)  the  attendance  by  the  Cal  span  SHE  for 
two  months  of  the  E-2C  NFO  training;  and  (2)  the  use  of  direct  observations 
of  task  performance  in  ground-based  devices  (Calspan,  E-2C) . It  is  also  of 
interest  to  note  that  it  was  necessary  that  the  contractor  redo  a previous 
government-performed^  task  analysis  in  one  of  the  programs  (Courseware, 
SH-2F). 

It  is  clear  that  SME  input  was  required  in  the  generation  of  operator 
task  lists,  but  there  were  differences  in  the  extent  to  which  SME  input  was 
provided  by  contractor  SMEs  as  opposed  to  Navy  SMEs.  Generally,  Courseware 
relied  relatively  more  heavily  on  the  Navy  SME,  while  Grumman  and  Calspan 
relied  relatively  more  on  the  contractor  SME.  Related  to  this  was  the  role 
of  the  instructional  psychologist  or  technologist.  Courseware  seemed  to 
use  such  personnel  more  extensively  in  articulating  the  task  data  deriving 
from  Navy  SME  input,  while  the  company  SME  for  the  other  two  contractors 
took  a more  direct  role  in  stating  the  task  data.  While  these  differences 
reflect  different  ISD  team  organizations,  it  is  likely  that  the  output  of 
this  phase  of  the  ISD  process  was  much  the  same  regardless  of  which  approach 
was  used.  The  critical  factor  was  that  the  SME  have  an  appropriate  exper- 
ience base. 

Format  and  Level  of  Detail 

The  actual  task  listings  were  not  available  for  the  present  review,  with 
the  exception  of  the  SH-2F  program,  so  comparisons  of  format  and  detail  are 
difficult.  Also,  it  is  difficult  to  separate  the  format  and  detail  of  the 
task  listing  from  that  developed  during  subsequent  task  analysis  activities 
such  as  the  preparation  of  objectives  hierarchies.  In  some  instances,  the 
initial  listing  appears  to  have  gone  to  a very  detailed  level,  while  in 
others  the  initial  listing  was  at  a gross  level  with  further  detail  devel- 
oped later. 


11/The  task  analysis  supplied  the  contractor  was  judged  to  be  inadequate 
in  that  it  devoted  great  attention  to  minute  details  of  equipment  manipu- 
lation while  ignoring  higher  level  distinctions  among  major  job  components. 
In  order  to  make  the  task  data  useful  for  later  ISD  steps,  the  contractor 
had  to  perform  the  analysis  again. 
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On  the  A-6E  program  the  contractor  (Grumman)  final  report — refers  to 
"the  generation  of  thousands  of  task  statements."  Consequently,  a computer 
was  used  to  handle  the  data.  Grumman  used  three  different  descriptive 
levels:  (1)  major  mission  events;  (2)  the  tasks  comprising  the  events;  and 
(3)  the  steps  that  comprise  the  task.  For  example,  "Pre-Approach"  is  a 
mission  event;  "Contact  Approach  Control  Agency"  is  a task;  and  "Set  UHF 
Frequency  for  Approach  Control"  is  a step  within  that  task. 

Cal  span,  on  the  E-2C  program,  broke  the  data  into  mission  segment,  task, 
and  task  elements,  a categorization  similar  to  that  of  Grumman.  However, 
the  meaning  of  thes^^escriptors  differs.  To  illustrate  its  schema,  the 
Cal  span  final  report—^  describes  the  mission  segment  "Recovery."  One  task 
in  this  segment  is  labeled  "Day  IFR  anJ  All  Night  Carrier  Recoveries  (Case 
III)."  Task  elements  in  this  task  include  items  such  as  "Complete  Approach 
Checklist"  and  "Initiate  Penetration."  The  Calspan  task  element,  "Complete 
Approach  Checklist,"  would  appear  as  a task  in  Grumman's  schema,  with  the 
task  then  broken  into  steps.  Thus,  Cal  span's  "task"  level  is  probably  the 
equivalent  of  Grumman's  "mission  event"  level,  and  Grumnan's  "task"  would 
probably  appear  as  a "task  element"  in  Calspan's  listing. 

On  the  EA-6B  program.  Courseware  defines  task  as  "any  sequence  of  events 
which  has  a clearly  definable  beginning  and  end  point  or  which  produces  an 
observable  product."  Courseware  speaks  of  mission  phases,  tasks,  and  sub- 
tasks. In  the  conduct  of  an  ECM  mission,  "Launch"  would  be  a mission  phase, 
"Take-off"  would  be  a task,  and  "Identify  and  Respond  to  Any  Aircraft  System 
Malfunctions"  would  be  a subtask.  In  the  SH-2F  program.  Courseware  used  a 
similar  conception,  though  the  task  listing  shows  additional  elements  within 
subtasks . 

The  principal  conclusion  from  these  comparisons  is  that  while  termin- 
ology and  format  may  differ,  and  level  of  detail  at  this  initial  stage  in 
ISO  may  vary,  the  procedures  used  are  essentially  the  same.  The  task  state- 
ments are  sooner  or  later  to  be  refined  to  the  level  of  detail  required  to 

12? 

-Campbell,  S.C.,  Feddern,  J.,  Graham,  G.,  and  Morganlander,  M.  A-6E 
Systems  Approach  to  Training  Phase  I Final  Report.  Technical  Report 
NAVTRAEQUIPCEN  75-C-0099-1.  Naval  Training  Equipment  Center,  Orlando,  FL, 
February  1977. 

^^Sugarman,  R.C.,  Johnson,  S.L.,  Mitchell,  J.F.,  Hinton,  W.M.,  and 
Fishburne,  R.P.  E-2C  Systems  Approach  to  Training:  Phase  I . Technical 
Report  NAVTRAEQUIPCEN  ^5-C-OlOl-l.  Nava 1 Tra i nTng  Tqui pment  Center,  Orlando, 
FL.,  December  1976. 
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develop  specific  behavioral  statements  of  objectives  and  to  allow  analysis 
of  the  essential  requirements  for  their  instruction. 

Task  Validation 

Once  the  initial  task  listing  was  developed,  the  programs  moved  to  some 
sort  of  validation  or  confirmation  of  the  listing.  Task  validation  is  im- 
portant in  ISO  because  the  task  listing  (or,  really,  the  tasks  selected 
from  the  task  listing)  forms  the  basis  for  all  subsequent  training  develop- 
ment activities.  Therefore,  it  is  essential  that  the  listing  include  all 
significant  tasks  to  be  performed  on  the  job  and  that  it  not  include  tasks 
that  are  not  required  to  be  performed  on  the  job.  Failure  on  the  first 
count  results  in  training  programs  that  are  deficient  in  preparing  grad- 
uates for  the  job,  while  failure  on  the  second  results  in  programs  that 
contain  unnecessary  material  and  are  unduly  costly. 

The  validation  procedures  described  in  the  project  reports  varied  some- 
what, Grumman  refers  to  comments  and  suggestions  from  a new  group  of  A-6E 
SMEs  (actually  one  pilot  and  one  bombardier/navigator)  who  joined  the  ISO 
team  after  the  task  listing  was  completed.  They  state  that  this  review 
was,  in  effect,  a "validation"  of  the  work  accomplished. 

Calspan  makes  no  specific  reference  to  task  validation  in  the  E-2C  pro- 
gram, other  than  to  a series  of  SME  interviews  "utilized  at  various  times 
during  data  analysis  to  ensure  that  the  data  base  was  complete  and  valid." 

In  contrast.  Courseware,  on  both  the  EA-5B  and  SH-2F  programs,  conducted 
a job  analysis  survey.  Data  were  sought  from  operational  aircrew  personnel 
concerning  (a)  frequency  of  occurrence  of  the  task,  (b)  how  soon  task  per- 
formance is  required  after  fleet  assignment,  and  (c)  where  the  task  was 
learned.  In  the  EA-6B  program  such  data  were  analyzed  from  17  pilots  and 
38  MFOs.  The  data  were  felt  by  the  contractor  to  be  adequate.  In  the 
SH-2F  program,  however,  results  were  judged  by  the  contractor  to  be  of 
questionable  use  due  to  failure  of  Navy  SMEs  to  complete  questionnaires 
properly  or  failure  to  meet  time  schedules.  The  SH-2F  program  report  notes 
questionnaires  not  filled  out  with  care,  random  marking  of  answers,  and 
the  like. 

Of  the  three  contractors,  only  Courseware  has  made  explicit  the  details 
of  a validation  procedure.  While  there  can  be  problems  in  carrying  out  job 
surveys  among  experienced  SMEs  or  job  incumbents  when  the  survey  procedure 
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Is  inadequate  (as  was  the  case  with  the  SH-2F),  most  fonnalized  ISO  systems 
emphasize  some  form  of  task  validation  as  an  important  and  necessary  step. 
While  both  the  A-6E  and  E-2C  programs  gave  some  recognition  to  this  fact, 
neither  treated  task  validation  as  a formalized  or  systematic  procedure. 

There  may  be  operational  difficulties  in  its  implementation,  but  task 
validation  is  desirable  and  feasible  in  an  ISD  project. 

Level  of  Effort 

Comparing  level  of  effort  over  the  various  programs  that  was  devoted 
to  the  task  listing/validation  activity  is  difficult.  As  has  been  noted, 
the  level  of  detail  represented  at  this  stage  varied,  and  the  manner  in 
which  Navy  SMEs  were  utilized  varied.  As  well  as  can  be  determined,  however, 
the  number  of  man-hours  devotedM/to  activities  in  the  task  listing/valida- 
tion step  are  shown  in  Table  3. 


TABLE  3.  EFFORT  DEVOTED  TO  TASK  LISTING/VALIDATION 


Program 

Contractor  Professional 
Effort® 

Navy  SME 
Effort® 

Total 

A-6E 

3,654 

3,654* 

E-2C 

1,740 

40 

1,780 

EA-6B 

445 

247 

692 

SH-2F 

480 

640 

1,120 

Effort  in  man-hours 

^’indeterminate  portion  of  1 ,348  hours  of  Navy  SME  effort  was 
probably  devoted  to  this  activity. 

As  can  be  seen  in  Table  3,  the  total  effort  devoted  to  task  listing/ 
validation  varied  greatly  over  the  four  programs.  Direct  comparisons  of 
these  data  are  difficult  because  of  the  differing  degree  of  detail  or  depth 


-^As  explained  later  in  the  General  Program  summaries  portion  of  this 
section  of  the  report,  Grumman  reported  both  "estimated  level  of  effort" 
and  "actual  level  of  effort"  for  their  various  activities.  All  data  dis- 
cussed here  with  reference  to  ISO  steps  for  the  Grumman  program  are  based 
on  their  "actual  level  of  effort"  reported.  Conversions  for  levels  of  effort 
originally  reported  in  man-months  were  made  on  the  assumption  that  one  man- 
month  equals  174  man-hours.  If  original  data  were  given  in  man-weeks,  a 
40-hour  week  was  assumed.  In  this  fashion,  all  programs  were  converted  to 
the  common  man-hour  metric. 
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to  which  ti^'-  task  listing  was  carried,  likely  differences  in  availability 
of  existing  cask  data,  differences  in  number  of  crew  positions  analyzed, 
and  differences  in  numbers  of  tasks  characterizing  a crew  position.  In 
general,  it  would  appear  that  the  EA-6B  effort  differs  from  the  other  three 
in  terms  of  both  amount  and  proportion  of  effort  devoted  to  the  task  listing/ 
validation  step.  The  EA-6B  program  shows  the  least  effort  in  this  area  of 
the  four,  a fact  that  would  seem  contradictory  to  the  fact  of  its  being  the 
only  one  of  the  programs  with  a systematic  task  validation  effort.  Pro- 
portionately, only  about  7%  of  the  contractor's  total  professional  effort 
was  devoted  to  task  listing/ validation  on  the  EA-6B  program,  whereas  approxi- 
mately 31%,  26%,  and  14%  of  total  contractor  professional  effort  was  devoted 
to  this  step  in  the  A-6E,  E-2C,  and  SH-2F  programs,  respectively.  The 
reasons  for  these  differences  could  not  be  ascertained,  though  as  will  be 
noted  later.  Courseware  had  a substantial  number  of  professional  man-hours 
on  the  EA-6B  effort  that  could  not  be  ascribed  to  specific  ISO  steps.  Some 
of  this  time  may  have  been  devoted  to  the  task  listing/validation  step. 

While  the  preceding  data  are  suggestive  of  some  differences  among  the 
three  contractors  in  the  relative  emphasis  placed  on  this  step  of  the  ISO 
process,  as  has  been  noted,  a variety  of  factors  may  have  produced  such  a 
result.  Most  likely,  the  differences  between  Grumman  and  Calspan,  on  the 
one  hand,  and  Courseware  on  the  other,  reflect  primarily  differences  in  the 
level  of  task  detail  developed  in  this  initial  step,  rather  than  in  emphasis 
per  se.  However,  the  approach  utilized  by  Courseware  was  more  procedur- 
al i zed  and  placed  relatively  greater  reliance  on  the  efforts  of  Navy  SME 
personnel,  so  there  probably  is  some  difference  between  the  contractors' 

ISO  procedures  in  this  area. 

TASK  SELECTION 

Given  the  definition  of  the  job  in  terms  of  its  required  tasks  devel- 
oped in  the  previous  activity,  the  next  step  in  the  ISO  process  involves 
selecting  some  subset  of  those  tasks  for  inclusion  in  the  training  program. 

In  most  advanced  training  courses  many  of  the  skills  and  knowledges  re- 
quired on  the  job  are  already  possessed,  in  whole  or  in  part,  by  the  trainees 
before  they  enter  the  training  program  of  concern  to  the  ISO  team.  There- 
fore, a specific  selection  of  tasks  to  form  the  basis  for  the  training 
program  must  be  made  in  view  of  the  job  requirements  and  the  trainees' 
entry  level  skills  and  knowledges. 


I 
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Method 

Much  of  the  data  or  information  on  which  task  selection  was  based  was 
developed  in  the  previous  task  validation  activity.  In  fact,  it  is  diffi- 
cult to  separate  the  two  activities  in  several  of  the  programs.  In  the 
more  formalized  expositions  of  the  ISD  procedure,  task  selection  is  usually 
based  on  factors  such  as: 

t Trainee  entry  repertoire 

• Task  frequency  on  the  job 

I Proportion  of  new  graduates  performing  task 

• Task  criticality  to  mission 

• Safety  considerations 

• Task  difficulty 

f Time  to  first  performance  on  job 

• Opportunity  for  training  on  the  job 

Of  the  four  programs  of  concern  here,  only  the  two  performed  by  Course- 
ware specify  an  explicit  procedure  for  selecting  tasks  for  training.  Their 
procedure  is  expressed  in  algorithmic  form  and  utilizes  as  input  data  the 
results  of  the  previously  described  job  analysis  survey  used  in  the  task 
validation  process.  Generally,  the  algorithm  used  in  the  Courseware  pro- 
grams operates  on  the  basis  of  answers  to  three  questions: 

1.  Did  more  than  20%  perform  the  task  on  most  missions? 

2.  Is  the  task  critical  to  the  mission? 

3.  Did  more  than  75%  learn  the  task  prior  to  RAG  training? 

If  the  answer  to  Q.l  is  YES,  Q.3  is  asked;  if  Q.3  is  YES,  the  task  is 
selected  to  "teach  in  review  mode,"  while  if  Q.3  is  answered  NO,  the  task 
is  selected  for  "full  training."  If  the  answer  to  Q.l  is  NO,  Q.2  is  asked; 
if  the  answer  to  Q.2  is  YES,  the  task  is  selected  for  full  training;  if 
Q.2  is  NO,  the  task  is  selected  to  teach  in  the  "Fam",  or  familiarization, 
mode. 

One  advantage  of  the  algorithmic  expression  of  the  selection  decision 
process  is  that  it  permits  the  process  to  be  computerized  and  to  be  handled 
procedurally.  Also,  it  makes  the  selection  procedure  explicit  and  repeat- 
able.  The  question  arises,  of  course,  as  to  the  meaningful  ness  or  validity 
or  the  particular  algorithm  chosen  and  the  validity  of  the  input  data. 
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Obviously,  the  questions  asked  by  Courseware  here  are  only  a portion  of 
the  pertinent  questions  that  could  be  asked,  but  they  are  pertinent  and 
comprise  a manageable  procedure. 

While  the  Courseware  algorithm,  as  any  algorithm,  can  be  questioned 
on  the  above  basis,  it  is  of  interest  to  examine  how  it  worked  in  practice. 
As  has  been  previously  mentioned,  the  SK-2F  program  experienced  consid- 
erable difficulty  in  securing  usable  data  from  the  task  validation  survey. 
In  fact  the  final  report  of  that  program  states  that  the  survey  data  were 
so  late  that  "...  many  decisions  had  to  be  made  at  the  informal  level  and 
in  the  absence  of  questionnaire  data."  It  further  states  that  because  of 
the  inferiority  of  the  data,  "Through  a study  of  prior  training  of  the 
students  and  normal  fleet  instruction  as  it  could  be  described  by  the 
project  SME's,  it  is  felt  that  the  tasks  selected  for  training  constitute 
a list  which  will  coordinate  well  with  fleet  training  programs."  Thus, 
it  would  appear  that  task  selection  in  the  SH-2F  program  was  made  largely 
through  an  informal  judgmental  procedure  rather  than  by  the  algorithm. 

In  contrast,  on  the  EA-6B  program  the  algorithm  seems  to  have  been 
utilized  smoothly  with  none  of  the  difficulty  encountered  in  the  SH-2F 
program.  Task  selection  required  only  four  hours  of  personnel  time  in  the 
EA-6B  program  and  is  described  as  "an  almost  trivial  effort."  The  EA-6B 
report  goes  so  far  as  to  recommend  that  task  selection  not  be  retained  as 
a step  in  ISO.  Similarly,  in  the  SH-2F  final  report  the  author  describes 
task  selection  as  primarily  an  administrative  function  that  should  be 
handled  by  the  Navy. 

Further  examination  of  the  results  of  the  task  selection  process  in 
the  EA-6B  program  shows  that  only  2 of  172  pilot/navigator  tasks  and  2 of 
178  ECMO  (electronics  countermeasures  operators)  tasks  were  not  selected 
for  training.  Of  the  170  pilot/navigator  tasks  selected,  17  were  "Fam 
only,"  25  were  "review  only,"  and  128  were  for  "full  training."  Of  the  176 
ECMO  tasks  selected,  one  was  "Fam  only,"  9 were  "review  only,"  and  166  were 
for  "full  training."  Viewed  from  this  perspective,  task  selection  made  no 
real  difference  in  the  content  of  the  program. 

In  the  Grumman  A-6E  program,  a section  of  the  final  report  Is  entitled 
"Task  Selection."  In  it,  a procedure  involving  elaboration  of  the  task 
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listing  through  a "Task  Analysis  Record"  form  is  described.  On  this  form, 
for  each  task  a variety  of  factors  are  elaborated  including:  (1)  crewman 
performing  task;  (2)  where  training  is  given;  (3)  skills  and  knowledge 
required  by  task;  (4)  conditions  under  which  task  is  performed;  (5)  cues 
involved  in  performance;  (6)  aircraft  system  involved;  (7)  degree  of 
difficulty;  (8)  factors  in  task  difficulty;  (9)  task  criticality;  (10) 
factors  in  performance  measurement;  and  (11)  other  special  factors  which 
impact  training.  Some  of  these  factors  would  seem  useful  to  task  selection, 
but  Grumman  says  nothing  about  the  details  of  how  such  data  were  used  in 
task  selection,  if  at  all.  It  is  not  known  whether  all  or  only  a portion 
of  the  tasks  were  actually  selected  for  inclusion  in  the  subsequent  training 
program,  or  on  what  basis  selection  was  made.  Their  report  states  that  no 
attempt  was  made  to  assess  the  appropriateness  of  a task  for  Readiness 
Squadron  training  or  to  judge  entry  level  skills  of  trainees.  SME  input 
apparently  was  accepted  without  question. 

Calspan  also  refers  to  task  selection,  stating  that,  with  consultation 
from  Navy  SMEs,  they  "selected  tasks  for  documentation  that  had  some  ex- 
pectation of  occurrence  and  were  amenable  to  training."  They  also  refer  to 
trainees'  incoming  skill  levels,  but  no  explanation  is  given  as  to  how 
skill  levels  were  assessed  or  to  how  task  selection  was  made. 

It  would  appear,  then,  that  all  four  programs  were,  in  fact,  based  on 
the  initial  task  listing  determined  largely  by  SME  judgment.  No  real 
selection  of  tasks  was  made,  nor  is  there  any  real  assessment  of  trainee 
input  skill  levels.  The  only  articulated  task  selection  procedure  was 
that  of  Courseware,  but,  in  effect,  it  had  almost  no  operational  influence 
on  program  content. 

Level  of  Effort 

The  available  data  did  not  allow  determination  of  amount  of  effort 
devoted  to  task  selection  for  the  Grumman  and  Calspan  efforts.  Their  task 
selection  efforts  likely  were  included  in  the  times  previously  given  for 
task  listing/validation. 

Courseware  reported  only  four  hours  involved  on  the  EA-6B  program, 
while  on  the  SH-2F  program  a total  of  80  hours  of  contractor  professional 
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j time  and  80  hours  of  Navy  SME  time  are  shown  for  task  selection. 

|j  TASK  ANALYSIS  ( OBJECTIVES  HIERARCHIES) 

; Once  tasks  are  selected  for  training,  they  are  elaborated  in  terms  of 

their  component  behaviors,  conditions  and  standards  of  performance,  inter- 
task dependency  relationships,  learning  requirements,  and  similar  factors. 

They  are  generally  cast  in  the  form  of  specific  behavioral  objectives  and, 

I thus,  become  the  training  objectives  for  the  program.  The  supporting 

j skills  and  knowledges  required  for  task  performance  are  also  identified 

and  stated  as  training  objectives.  It  is  in  this  area  of  ISO  that  there 
I may  result  substantial  differences  in  the  content  of  training  programs,  for 

the  processes  of  determining  inter-task  sequential  learning  contingencies 
and  of  identifying  supporting  or  enabling  skills  and  knowledges  are  analy- 
tical and  judgmental  in  nature.  Consequently,  two  different  ISO  analysts 
can  come  up  with  quite  different  program  content  utilizing  the  same  input 
data.  Such  differences  impact  program  length,  media  and  resource  require- 
ments, and  program  cost. 

Method 

The  methods  employed  by  the  various  contractors  in  analyzing  tasks 
and  objectives  differed  somewhat.  The  goal  of  this  level  of  the  task 
analysis  for  all,  however,  was  to  develop  the  information  needed  for  struc- 
turing the  training  program,  for  selecting  media,  and  for  preparation  of 
the  lesson  specifications.  In  the  Grumman  A-6E  program,  the  factors  pre- 
viously cited  in  task  selection  (e.g.,  cues,  system,  difficulty,  criti- 
cality, etc.)  were  used  in  connection  with  the  development  of  specific 
behavioral  objective  (SBO)  statements.  Objectives  were  also  classified 
on  the  basis  of  eight  major  taxonomic  categories:  (1)  knowledge;  (2) 
comprehension;  (3)  discrimination;  (4)  application;  (5)  analysis;  (6) 
synthesis;  (7)  evaluation;  and  (8)  complex  performance.  The  SBOs  were 
documented  on  an  "Instructional  Systems  Design  Record"  form  that  gave  the 
task  identification  data,  condition/constraints,  performance  standard,  the 
SBO  statement  and  ID  number,  taxonomic  data,  a criterion  test  statement, 
and  test  type  and  format.  Nothing  appears  in  this  record  concerning 

o 
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hierarchical  or  inter-task  relationships,  but  hierarchical  relationships 
were  treated  in  the  development  of  instructional  strategies  and  sequences 
(see  discussion  in  a later  section  of  this  report). 

In  the  E-2C  effort  of  Calspan,  behavioral  objectives  were  developed 
from  the  task  data.  The  format  included  identifying  data  for  each  objec- 
tive, as  well  as  its  criticality,  difficulty,  conditions  and  standards. 

In  addition,  it  included  reference  to  concurrent  tasks  that  the  operator 
must  perform,  as  well  as  to  interaction  tasks  involving  other  crewmembers. 

As  with  the  previous  parts  of  the  ISD  procedure.  Courseware  had  the 
most  highly  articulated  system  for  analyzing  the  objectives.  The  tasks  are 
analyzed  in  order  of  their  complexity.  Subtasks  are  identified  until  entry 
level  behavior  is  reached.  The  hierarchical  relationships  among  tasks, 
subtasks,  and  elements  are  portrayed  graphically  in  a manner  that  shows 
the  interdependencies  among  objectives  that  underlie  a given  task.  This 
information  is  then  used  in  the  structuring  of  syllabi  and  lessons. 

In  developing  these  hierarchies.  Courseware  refers  to  "pruning"  (i.e., 
the  elimination  of  trivial  objectives);  to  "summarization"  (i.e.,  the  re- 
quirement that  the  student  state  a procedure  before  executing  it);  to  "be- 
havior approximation"  (i.e.,  not  performing  the  actual  task  when  dangerous, 
but  an  approximation);  and  to  "systems  introduction  hierarchy  structure," 
(i.e.,  the  requirement  that  the  student  be  familiar  with  basic  system, 
subsystem,  and  component  relationships). 

Thus,  while  there  are  similarities  over  the  programs  in  this  aspect 
of  their  task  analysis,  there  are  also  differences.  In  all  the  contractor 
systems,  the  analysis  is  subjective  ultimately,  and,  thus,  its  utility  is 
dependent  upon  the  skill  of  the  executor.  Standards  of  performance  are 
subjective,  and  in  all  four  of  these  programs  they  seem  to  be  based  upon 
a somewhat  uncritical  acceptance  of  SME  input,  NATOPS^^or  similar  sources. 
Similarly,  assumed  hierarchical  relationships  between  cognitive  objectives 
and  task  performance  are  based  on  SME  input.  As  has  been  noted,  differ- 
ences in  the  assumptions  made  by  the  course  developer  during  task  analysis 
have  potentially  large  consequences  in  terms  of  subsequent  program  costs. 

This  aspect  of  ISD  is  still  somewhat  more  art  than  science  and  is  worthy 

^ The  question  raised  here  is  not  the  appropriateness  of  NATOPS  standards 
to  fleet  operations,  but  their  appropriateness  as  training  program  standards. 
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of  research  investigation,  for  it  has  major  consequences  for  ultimate  pro 
gram  cost  effectiveness. 

Level  of  Effort 

As  has  already  been  noted,  the  available  data  do  not  lend  themselves 
well  to  comparison  over  the  four  ISO  programs  at  this  level  of  detail. 

However,  Grumman  shows  3,045  of  its  total  11,658  professional  hours  (26%) 
going  to  development  of  SBOs . On  the  E-2C  program,  Calspan  shows  965  of 
6,581  professional  hours  (15%)  devoted  to  development  of  training  objec- 
tives. On  its  EA-6B  program.  Courseware  shows  736  of  6,135  professional 
hours  (12%)  plus  208  of  1,547  Navy  SME  hours  (13%)  devoted  to  hierarchy 
analysis.  On  the  SH-2F,  1,044  of  the  contractor's  3,469  professional  hours  ! 

(30%)  and  1,392  of  5,752  Navy  SME  hours  (24%)  were  devoted  to  objectives 

hierarchy  development.  Thus,  all  four  programs  show  substantial  amounts  * 

of  time  devoted  to  objectives  and  hierarchies.  It  is  a time  consuming  j 

activity  in  the  overall  ISO  process.  i 

MEDIA  SELECTION 

The  importance  and  the  cost  of  media  in  modern  aircraft  training  programs 
have  risen  steadily  in  recent  times.  For  many  years,  flight  training  pro- 
grams relied  principally  on  flight  instruction  in  the  aircraft  supported 
by  ground  training  consisting  principally  of  stand-up  lectures,  with  minor 
training  aid  assistance,  and  use  of  standard  textbooks  and  flight  manuals. 

Often,  the  relationship  between  ground  training  and  that  which  it  supported 
was  of  dubious  nature.  One  thing  ISO  has  purported  to  do  is  to  make  the 
relationship  more  truly  a supportive  one.  Another  thing  it  seems  to  have 
done,  perhaps  inadvisedly,  is  to  stress  the  development  of  a multiplicity 
of  media  to  support  instruction. 

In  part,  this  increase  in  emphasis  on  a variety  of  media  has  resulted 
from  the  advantages  assumed  (on  the  basis  of  considerable  research)  to 
accrue  from  individualization  and  proficiency  pacing  of  instruction.  It 
has  also  derived  from  the  developments  that  have  occurred  in  instructional 
technology,  developments  that  have  related  media  types  to  instructional 
objectives  types  in  terms  of  instructional  or  learning  efficiency.  As  a 
result  of  these  factors  and  the  tendency  of  ISD  systems  to  emphasize 
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hierarchical  structure  and  "cogni ti ve-before-performatory"  objective 
sequences  (e.g..  Courseware's  "summarization"  and  "systems  introduction" 
features),  the  use  of  various  media  has  grown  markedly  in  flight  training 
programs . 

The  major  media  development  of  recent  years  has  been  the  modern 
digital  flight  simulator.  But,  programmed  texts,  sound-slide  presenta- 
tions, ETV,  CAI,  motion  pictures,  workbooks,  learning  centers,  and  other 
media  have  received  emphasis  also.  Thus,  it  is  not  surprising  that 
selection  of  media  is  an  area  of  considerable  concern  in  ISO,  particu- 
larly when  one  considers  the  fiscal  investment  that  even  the  simpler  media 
can  represent.  Not  only  can  initial  investments  in  media  be  substantial, 
the  maintenance  and  updating  costs  for  media-centered  programs  can  be 
substantial.  Such  factors  must  be  borne  in  mind  by  the  course  developer, 
along  with  concern  for  instructional  effectiveness,  as  media  selection 
decisions  are  made. 

Method 

The  procedure  used  for  selecting  media  to  accomplish  the  training 
program's  objectives,  of  course,  has  much  to  do  with  the  characteristics 
of  the  ensuing  program.  In  the  present  ISO  efforts,  media  selection  was 
constrained  by  a number  of  factors,  perhaps  the  principal  of  which  was  its 
restriction  generally  to  currently  available  or  planned  simulators  or  WSTs 
for  the  aircraft  of  concern.  Thus,  the  media  selections  can  be  considered 
optimal  only  in  the  sense  that  they  were  optimized  for  the  media  already 
available. 

In  the  Grurman  A-6E  program,  media  requirements  for  each  SBO  were 
examined  with  reference  to  three  phases  of  instruction:  (1)  initial  in- 
struction; (2)  practice;  and  (3)  demonstrati on/ test.  They  were  examined 
for  each  of  these  phases  in  terms  of  two  factors--stimulus  characteristics 
necessary  to  present  the  materials,  and  capabilities  required  for  optimum 
response.  Just  how  this  analysis  was  done  could  not  be  determined  from 
the  available  information,  but  it  was  presumably  a judgmental  process  by  the 
educational  specialist  on  the  project.  In  any  event,  the  WST  and  proce- 
dures trainers  receive  heavy  emphasis  in  the  Grumman  analysis,  as  do  sound- 
slide  presentations  and  overhead  transparencies. 
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Several  media  factors  distinguish  the  A-6E  program  from  the  others. 

One  is  the  use  of  the  TC-4C  aircraft  which  is  an  "inflight  device"  with 
several  bombardier/navigator  stations.  In  justifying  the  use  of  the  TC-4C 
aircraft  as  a trainer,  Grumman  states  that,  while  many  of  the  SBOs  might 
be  taught  as  well  in  the  WST,  "the  TC-4C  provides  valuable  in-flight  demon- 
stration of  the  capability  of  the  B/Ns  to  use  the  operational  equipment 
under  actual  flight  conditions." 

Another  media  point  of  interest  is  Grumman' s suggestion  that  "paper 
simulations"  be  developed  so  that  the  students  can  better  learn  and  utilize 
"headwork."  By  headwork  is  meant  the  ability  to  generalize  skills  to  new 
situations.  While  no  SBOs  were  specifically  assigned  to  this  medium,  it 
is  an  interesting  suggestion  that  is  worth  exploring  in  view  of  the  great 
stress  on  decision  making  skills  in  modern  weapons  systems. 

The  report  of  Calspan's  E-2C  program  does  not  give  extensive  details 
of  the  media  selection  process.  Note  is  taken  of  the  emphasis  on 
use  of  available  media.  Also,  reference  is  made  to  selecting  media  that 
allow  (1)  student-paced  instruction,  (2)  individualized  presentation,  and 
(3)  immediate  knowledge  of  results.  Consequently,  Cal  span  settles  on  the 
general  purpose,  audio-visual  carrel  to  carry  the  program,  along  with  avail- 
able ground-based  simulators  and  training  devices.  Of  interest  is  their 
suggestion  that  the  CPT  be  modified  to  permit  use  of  the  same  slide-tape 
programs  used  in  the  carrel.  While  the  carrel  largely  replaced  classroom 
instruction,  some  classroom  instruction  is  deliberately  retained  as  a 
transition  phase  and  to  tell  "sea  stories." 

Courseware  handles  media  selection  in  a much  more  extensive  and  pro- 
ceduralized  manner,  relying  on  an  algorithm,  exercise  of  which  will  lead 
to  one  of  some  44  media  decisions.  While  there  are  44  different  pathways, 
some  of  their  end  points  are  identical.  Most  end  points  list  first  and 
second  choice  alternatives,  and  sometimes  more.  The  media  selection  al- 
gorithm is  applied  to  each  training  objective.  Input  data  for  the  algorithm 
are  answers  to  five  questions.  These  questions  and  their  possible  answers 
are  as  follows: 
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a.  Familiarization 

b.  Discriminated  recall 

c.  Rule  using 

2.  What  is  the  level  of  content  to  be  taught? 

a.  Familiarization 

b.  Paired  associate 

c.  Concept 

d.  Rule 

3.  What  is  the  size  of  the  minimum  set  of  critical  instances  needed? 

a.  Small 

b.  Large 

4.  What  is  the  minimum  display  requirement? 

a.  Verbal  and/or  symbolic  and/or  static  simple  pictorial 

b.  Verbal  and/or  symbolic  and/or  static  complex  pictorial 

c.  Dynamic  pictorial 

d.  Interactive 

5.  How  large  is  the  memorization  component? 

a . Sma 1 1 

b.  Large 

The  answers  to  these  five  questions  can  be  assigned  by  the  instruc- 
tional technologist  for  each  objective,  and  the  process  of  media  selection 
can  then  be  automated.  In  the  EA-6B  program,  this  allowed  the  development 
of  four  alternative  media  plans  which  were  then  evaluated  in  terms  of 
relative  time,  personnel,  and  material  costs.  The  media  plans  differed 
in  terms  of  requirements  for  acquiring  new  media  such  as  CAI , projectors, 
etc.  The  contractor  recommended  a media  plan  for  the  EA-6B  that  required 
CAI  (one  that  would  involve  new  procurement),  as  well  as  the  existing  WST, 
tactics  trainer,  and  other  devices. 

While  no  problems  in  applying  the  media  selection  model  were  noted 
in  the  EA-6B  report,  for  the  SH-2F  program  problems  were  again  evident. 
Questions  as  to  availability  of  funds  to  support  media  costs  apparently 
resulted  in  much  lost  motion  in  the  SH-2F  program  media  selection  and 
resulted  in  the  necessity  for  making  "final"  selection  decisions  several 
times. 
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Level  of  Effort 

The  time  accounting  on  the  Calspan  E-2C  program  did  not  separate  media 
selection  from  other  aspects  of  the  development  of  instructional  objec- 
tives and  preparation  of  lesson  specifications.  Grumman  does  break  media 
selection  out  on  its  A-6E  program,  showing  a total  of  1,392  (12%)  of 
11,658  total  professional  man-hours  for  this  function.  On  the  Courseware 
EA-6B  program,  152  (2%)  of  6,135  professional  man-hours  were  devoted  to 
media  selection,  while  on  its  SH-2F  program  320  (9%)  of  the  total  3,469 
professional  man-hours  were  involved.  No  Navy  SME  assistance  was  noted 
on  any  of  the  programs  for  this  function. 


COURSE  DESIGN 

Somewhere  in  the  sequence  of  ISD  events,  the  ISD  analyst  must  make 
decisions  as  to  the  general  structure  of  the  course  and  the  manner  in 
which  various  portions  of  the  instruction  will  be  sequenced.  It  is  largely 
to  this  end  that  the  information  on  objectives  hierarchies,  previously 
discussed,  was  developed.  Not  only  must  consideration  be  given  to  inter- 
objective and  inter-task  relationships,  it  must  also  be  given  to  inter- 
media relationships.  Instructional  strategies  and  sequences  are  adopted, 
and  they,  in  turn,  provide  the  organizing  structure  for  the  course.  These 
may  include  concepts  such  as: 

• Maximize  hands-on  instruction 

• Teach  first  in  the  least  expensive  medium 

• Teach  from  simple  to  complex 

• Introduce  difficult  material  early 

• Relative  emphasis  of  massed  vs.  distributed  practice 

• Introduce  system  structure  before  operation 

• Individualization 

• Self-pacing 

I Proficiency-pacing 

The  choices  and  decisions  made  in  this  area  have  a profound  effect 
on  the  characteristics  of  the  training  program,  its  effectiveness,  its 
costs,  and  its  efficiency.  As  was  noted  in  the  previous  discussions  of 
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task  analysis  and  media  selection,  some  substantial  differences  among 
different  ISD  practitioners  may  arise  at  this  point. 

Method 

The  following  exposition  of  the  methods  used  in  structuring  the  four 
training  programs  will  touch  only  the  major  aspects  of  course  structure. 
The  particular  structure  adopted  in  a program  then  determines  the  nature 
of  the  lesson  specifications  that  implement  that  structure.  At  a gross 
level,  all  four  programs  followed  a similar  organization,  but  at  a finer 
level  there  were  differences. 

A-6E . In  devising  its  general  approach  to  A-6E  training,  Grumman 
chose  to  organize  Pilot  training  around  a phase  of  flight  and  phase  of 
mission  sequencing,  whereas  for  the  Bombardier/Navigator  (B/N)  training 
they  organized  around  a system/subsystem  orientation.  They  emphasize  five 
guidelines  or  strategies  in  their  program.  These  are:  (1)  provide  early 
hands-on  practice;  (2)  preserve  task  integrity;  (3)  consider^  the  prin- 
ciples of  massed  vs.  distributed  practice;  (4)  introduce  high  skill  tasks 
early  to  allow  more  practice;  and  (5)  maximize  use  of  "real"  hardware  to 
enhance  transfer. 

A-6E  Pilot  SBOs  were  initially  grouped  into  six  training  phases: 
Familiarization;  Visual  Weapons  Delivery;  Tactics;  Navigation;  System 
Weapons  Delivery;  and  Carrier  Qualification.  The  Bombardier/Navigator 
training  phases  initially  were  seven  in  number:  Familiarization;  System/ 
Subsystem  Design,  Theory,  and  Application;  Airframe  Emergencies  and  System 
Malfunctions;  Navigation  Theory  and  Application;  Attack  Theory  and  Appli- 
cation; Air/Combat  Theory  and  Application;  and  Takeoff  and  Landing  Checks. 
The  training  phases  were  then  divided  into  units,  which  were,  in  turn, 
divided  into  lessons.  These  groupings  were  then  reviewed  by  Navy  SMEs  and 
revised  as  appropriate.  This  revision  resulted  in  a Pilot  training  pro- 
gram of  six  phases  totaling  12  units,  while  the  B/N  program  consisted  of 
four  phases  totaling  10  units.  Media  recommendations  were  reviewed,  and 
media  consolidations  were  made  to  reduce  the  number  of  different  media 

'^The  report  gives  no  indication  of  how  these  principles  were  consid- 
ered or  used. 
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used  within  a given  lesson  to  a i"ore  manageable  number.  Lessons  (which 
might  be  further  subdivided  into  sessions)  were  then  sufficiently  defined 
to  proceed  with  the  next  step  in  the  process,  that  of  lesson  specifica- 
tions. A total  of  55  lessons  resulted,  19  of  which  were  for  Pilot  only, 

17  for  the  B/N  only,  and  19  were  common  to  Pilot  and  B/N  training. 

.E^2C.  Calspan  distinguishes  within-blocks  (lessons)  instructional 
strategy  from  be tween-blocks  (lessons)  strategy.  The  basic  strategy 
within  cognitive  instructional  blocks  involved  student-paced  instruction, 
individualized  presentation,  and  testing  with  immediate  knowledge  of  re- 
sults. For  the  practice  or  sortie  (as  opposed  to  cognitive)  blocks,  the 
strategy  emphasized  hands-on  learning  with  immediate  feedback.  Pacing 
of  instruction  would  depend  on  instructor  evaluation  of  performance. 

The  between-blocks  strategy,  i.e.,  the  sequencing  of  blocks,  involved 
early  hands-on  experience,  attention  to  enabling  skills  prior  to  their 
use  in  complex  skills,  and  the  interspersal  of  hands-on  training  with  cog- 
nitive training.  With  reference  to  this  last  point,  Calspan  favors  in- 
tegration of  devices  and  aircraft  throughout  the  training  sequence,  rather 
than  a sequential  completion  of  cognitive,  device,  and  aircraft  training. 

Distributed  practice  is  preferred  in  their  schema  to  massed  practice.  Other 
important  factors  in  the  E-2C  program  sequencing  were  difficulty,  criti- 
cality, and  frequency  of  task  occurrence.  These  latter  factors,  it  is 
presumed,  are  based  on  some  pooling  of  contractor  and  Navy  SME  judgment, 
though  the  details  are  not  made  explicit.  In  general,  initial  training  is 
with  the  simplest  medium  (carrel  or  classroom),  with  subsequent  practice 
occurring  in  the  devices  and  aircraft. 

Both  Pilot  and  NFO  courses  in  the  E-2C  program  are  structured  around 
a "systems  within  mission  context."  By  this  is  meant  that  systems  knowl- 
edge instruction  is  integrated  into  mission  training.  Lessons  are  organ- 
ized into  modules  on  the  basis  of  general  content.  The  graphic  schema 
for  showing  lesson  and  module  structure  allows  distinction  of  those  lessons 
that  must  be  taught  in  a given  sequence,  on  the  basis  of  hierarchical 
relationships,  and  those  for  which  ordering  is  optional.  The  resulting 
E-2C  P/CP  course  has  19  modules  totaling  92  lessons,  while  the  NFO/FT 
course  has  38  modules  and  137  lessons. 
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EA-6B.  The  sequencing  of  instruction  for  the  EA-6B  program  was  a 
relatively  simple  procedure  because  of  the  considerable  detail  developed 
by  Courseware  personnel  in  previous  ISO  activities.  Instructional  units 
were  formed  by  taking  objectives  relatively  high  in  the  hierarchy.  A 
unit  then  was  defined  as  the  objectives  subordinate  to  that  level.  Lessons 
within  units  were  structured  around  medium  complexity  level  objectives. 
Sequences  of  objectives  within  lessons  were  then  determined.  In  addition 
to  objectives  hierarchical  relationships,  factors  in  sequencing  included 
early  hands-on  training  and,  where  independent  of  the  hierarchy,  more 
critical  or  difficult  objectives  being  presented  first.  This  process  was 
carried  out  by  contractor  specialists  working  closely  with  the  Navy  SMEs. 
Following  this,  the  unit  and  lesson  structure  was  integrated  with  the  media 
selections.  Generally,  media  needs  were  consolidated  so  as  to  identify  a 
single  optimum  medium  for  each  lesson. 

The  next  activity  involved  development  of  unit  and  lesson  maps.  These 
maps  are  actual  diagrams  of  the  instructional  sequence  along  with  verbal 
exposition  of  the  task  or  action,  frequently  with  conditions  and  standards 
stated.  The  maps  and  objectives  were  then  reviewed  by  an  SME  not  involved 
with  their  development. 

Courseware  notes  four  problems  that  occurred  during  this  process,  though 
all  were  apparently  handled  without  great  difficulty.  The  first  was  the 
fact  that  the  training  squadron  had  certain  sequencing  preferences  that  were 
independent  of  the  objectives  hierarchies  and  could  not  be  deduced  there- 
from. Identification  of  such  preferences  (particularly  if  they  are  to  be 
deterministic  constraints)  should  be  made  early  in  the  ISO  effort  to  avoid 
having  to  redo  the  sequencing  several  times.  The  second  area  concerned 
the  interaction  of  sequencing  and  media  selection.  Compromises  in  optimum 
media  or  in  optimum  (hierarchical)  sequencing,  or  both,  must  be  made.  The 
third  problem  was  that  the  time  to  complete  the  unit  and  lesson  maps  was 
greater  than  expected.  The  fourth  problem  was  that  SMEs  were  not  readily 
available  to  review  certain  products,  resulting  in  delays  in  completion. 


SH-2F.  The  procedure  used  by  Courseware  in  the  SH-2F  program  was  much 


NAVTRAEQUIPCEiN  77-C-0009-1 

the  same  as  that  described  for  the  EA-6B  program.  However,  several  addi- 
tional considerations  are  described  that  are  of  some  Interest.  The  general 
course  organization  involved  a combination  of  missions,  phases  of  flight, 
and  equipment  as  organizing  factors.  Lessons  made  up  units,  and  groupings 
of  units  were  labeled  as  "sections."  Sections  were  intended  to  require 
no  more  than  two  or  three  weeks  for  their  accomplishment,  whereas  a unit 
was  to  be  no  longer  than  one  week.  In  turn,  lessons  were  to  require  no 
longer  than  one  day  for  their  completion. 

Factors  in  defining  units  were  comprehensiveness  of  scope  (i.e.,  the 
unit  should  contain  all  lessons  on  a given  topic)  and  the  presence  of  a 
terminal  testing  point.  Definition  of  lessons  was  based  on  logical  test 
points,  attention  to  the  one  day  time  limitation,  and  "nontriviality"  of 
lessons.  Major  factors  in  the  sequencing  of  units  and  lessons  included 
resource  requirements,  recycle  time  for  failed  students,  criticality  and 
difficulty  of  the  tasks/behaviors  involved. 

Level  of  Effort 

It  was  not  possible  to  isolate  the  amount  of  effort  devoted  to  the 
strategy/sequencing  function  for  the  A-6E  and  E-2C  programs.  On  the 
Courseware  EA-6B  program,  662  contractor  professional  man-hours  and  97 
Navy  SME  hours  were  reported,  while  on  the  SH-2F  program  the  totals  were 
240  contractor  hours  and  160  Navy  SME  hours.  While  these  totals  repre- 
sent a relatively  small  part  of  the  overall  program  totals,  the  effort 
devoted  to  this  activity  is  not  inconsequential. 

LESSON  SPECIFICATIONS 

The  Phase  I programs  were  expected  to  produce  a complete  set  of  lesson 
specifications.  The  Navy  RFP  stated  on  p.6  of  its  Specification  for  Systems 
Approach  to  Training,  "...  The  following  results  are  expected  from  this 
study:  a.  A complete  set  of  lesson  specifications  shall  provide  the  basis 
for  the  follow-on  production  work  and  program  implementation.  The  lesson 
specifications  will  be  the  guidelines  that  will  spell  out  in  detail  how 
each  lesson  should  be  organized  and  taught  and  what  its  content  should 
include."  Thus,  the  intent  was  co  document  the  instructional  program 
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concept  in  a form  such  that  someone  other  than  the  Phase  I contractor, 
at  least  potentially,  could  carry  out  Phase  II,  including  actual  authoring 
and  production  of  all  lesson  and  course  materials.  For  this  reason,  the 
lesson  specifications  would  have  to  define  what  was  to  be  taught,  when, 
and  how.  Otherwise,  the  Phase  II  effort  could  inject  new  material,  a 
» new  course  organization,  or  a new  system  design  concept  and  thereby  sub- 

stantially negate  what  had  been  done  in  Phase  I. 

Should  the  same  contractor  or  agency  perform  both  Phases  I and  II, 
the  problem  of  information  transfer  between  phases  would  be  less  critical, 
since  there  would  likely  be  some  personnel  continuity  over  the  two  phases. 
However,  since  these  four  ISO  efforts  were  conceived  by  the  Navy  as  po- 
tentially involving  different  executors  for  Phases  I and  II,  the  question 
of  the  degree  to  which  the  lesson  specifications  can  effect  the  necessary 
information  transfer  (and,  in  fact,  enforce  the  system  design  concept)  is 
a very  critical  one.  The  lesson  specifications  constitute  the  principal 
products  of  these  Phase  I efforts.  If  the  lesson  specification  documents 
are  deficient  with  reference  to  the  information  transfer  and  system  concept 
enforcement  factors,  then  the  Navy's  management  concept  of  separable  Phase  I 
and  II  efforts  would  not  be  implementable. 

The  existence  of  effective  lesson  specification  documents,  as  assumed 
in  the  Navy's  schema,  by  no  means  guarantees  an  effective  or  good  training 
program  outcome.  If  the  prior  ISD  analytical  steps  were  improperly  per- 
formed, or  if  the  training  system  concept  itself  is  deficient,  effective 
lesson  specifications  will  simply  make  it  highly  probable  that  the  concept 
will  be  carried  out  as  planned  (with  the  consequent  program  being  less  than 
optimal).  In  contrast,  ineffective  lesson  specifications  will  make  it 
unlikely  that  the  training  system  concept  (good  or  bad)  will  be  carried 
out  as  planned. 

Format 

Since  the  lesson  specification  is  conceived  as  the  basic  information 
transfer  means  between  Phases  I and  II,  its  format  is  important.  There 
were  some  significant  differences  among  the  three  contractors  in  the  format 
used,  so  format  will  be  discussed  in  some  detail. 
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A-^£,  The  lesson  specification  format  used  by  Grumman  is  comprised 
of  the  major  sections  shown  below. 

• Lesson  Number 

• Lesson  Title 

• Phase  Number  and  Title 

• Unit  Number  and  Title 

t Specific  Behavioral  Objectives  (No.  & Statement) 

• Media  (by  Session  for  Initial  Learning,  Practice,  & Demo/Test) 

• Lesson  Duration  (by  Classroom,  Practice,  & Demo/Test) 

t References 

• Lesson  Structure 

Each  lesson  contains  a number  of  SBOs,  and  lessons  may  involve  multiple 
sessions.  The  SBOs  vary  in  the  level  of  detail  and  the  magnitude  of  the 
task  described.  For  example,  they  vary  from  statements  such  as  "locate 
and  identify  the  X,"  "explain  the  relationship  between  A and  B,"  etc., 
to  "the  replacement  pilot,  when  ready  to  take  off,  shall  release  brakes, 
and  by  engaging  nosewheel  steering  and  manipulating  rudder  pedals  maintain 
center  of  runway,  call  off  airspeed  and  runway  remaining  information  to 
the  B/N  and  at  an  appropriate  speed  ease  back  on  control  stick,  etc." 

Primary  and  alternate  media  are  shown  for  each  session  within  the 
lesson  separately  for  initial  learning  (i.e.,  classroom),  practice,  and 
demo/test.  In  virtually  all  areas  the  initial  learning  is  in  the  class- 
room by  sound/slide,  overhead  transparency,  or  slide  media. 

The  section  entitled  "Lesson  Structure"  is  one  or  more  attached  pages 
that  contain  three  columns  of  boxes  in  which  are  listed  the  SBO  numbers 
covered  in  each  session.  The  three  columns  are  titled  "initial  learning," 

"practice,"  and  "demo/test."  In  addition  to  listing  the  SBO  numbers,  there 
is  given  a summary  statement  of  the  subject  of  those  SBOs,  e.g.,  "basic 
instruments  system  information  directly  related  to  After  Takeoff,  etc." 

Also  shown  in  each  box  are  the  primary  and  alternate  media.  No  information 

on  standards  of  performance  is  given  routinely,  though  some  SBO  statements  ? 

may  contain  or  imply  a standard.  The  lesson  structure  sheet  does  not 

contain  any  new  information  other  than  indicating  SBOs  by  session  and  the  1 


NAVTRAEQUIPCEN  77-C-0009-1 


sunmary  statement.  The  SBO  ordering  is  the  same  as  that  in  the  body  of  the 
lesson  specification,  and  the  media  listing  is  redundant. 

E-2C.  The  format  used  by  Calspan  for  its  lesson  specifications  treats 
the  following  topics  on  the  cover  sheet: 

• Lesson  Number 

• Module  Title 

• Lesson  Title 

• Type  (Cognitive,  Practice,  or  Sortie/Scenario 

e Training  Objectives 

• Time 

• Prerequisite  Lessons 

t Additional  Reference  Materials 

• Training  Devices 

For  cognitive  lessons,  a 100%  standard  of  performance  is  cited,  whereas 
for  practice  and  sortie  lessons  standards  are  stated  for  each  objective  in 
supplemental  sheets.  These  supplemental  sheets  for  each  objective  in  the 
lesson  depict  criticality  and  difficulty  ratings  (three-point  scale)  for  the 
objective,  and  they  then  present  the  task  element  behaviors  that  comprise 
the  objective.  These  sheets  next  give  the  performance  limits  and  initial 
conditions  for  the  objective. 

A "sequence  of  instruction"  sheet  follows  the  cover  sheet  for  all  cog- 
nitive objectives.  These  sheets  identify  in  hierarchical  fashion  suggested 
"teaching  points"  (enabling  objectives)  with  a description  of  associated 
media  support  (e.g.,  a picture  of  the  instrument  panel  with  a specific 
indicator  highlighted).  In  addition,  specific  notes  to  be  included  in  the 
presentation  are  shown.  The  intent  is  to  provide  information  from  which 
a sound/slide  presentation  could  be  developed. 

EA-6B  and  SH-2F.  The  lesson  specification  format  used  by  Courseware 
is  fairly  elaborate.  Also,  certain  of  the  terms  used  are  unusual  and  re- 
quire definition.  Each  general  specification  includes  the  following 
sections: 

• Lesson  Map 

• Lesson  Objectives 

• Lesson  Introduction 
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Thon,  for  each  segment  (objective)  within  the  lesson  the  following  are 
given: 

• Segment  Designation  (ID  for  course,  unit,  lesson,  and  segment) 

e Topic 

• Objective 

• Media 

• Level  of  Content 

• Level  of  Behavior 

• Generality  Statement 

• Figure  Numbers 

• Help 

t Instance  Spec  or  Practice  and  Testing 

• Special  Teaching  Points 

t Graphics  Specifications 

Then,  there  are  additional  Items  relating  to  attached  figures  or  illus- 
trations, security  classification,  and  Instance  specifications.  The 
Instance  specifications  must  include: 

• Type  Description 

• Format  Description 

• CEA  (Common  Error  Analysis) 

• MCS  (Minimum  Critical  Set  of  Instances) 

• Instance  Production 

• Testing  Criteria 

As  noted,  many  of  these  categories  require  some  definition,  while  for 
others  the  meaning  is  self-evident.  The  “lesson  map"  shows  graphically 
the  hierarchical  relationships  of  the  various  topic  areas  included  in  the 
lesson.  The  "lesson  introduction"  is  intended  to  explain  to  the  student 
why  he  is  studying  the  lesson  and  how  it  relates  to  other  information  and 
lessons.  "Segment"  is  simply  the  name  given  to  individual  objectives  in 
the  lesson.  "Level  of  content"  involves  three  basic  categories  of  content 
fact  (identities),  concepts,  and  rules.  "Level  of  behavior"  involves  four 
categories;  familiarization,  discriminated  recall,  classification,  and 
rule  using.  "Level  of  content"  and  "level  of  behavior"  relate  to  factors 
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in  Courseware's  system  for  defining  relationships  between  content/behavior 
level  and  instructional  strategies  and  media  requirements. 

The  "generality"  portion  of  the  specification  refers  simply  to  a state- 
ment of  the  facts  to  be  memorized,  the  attributes  of  a concept,  or  the 
steps  or  formula  of  a rule.  In  short,  it  is  a statement  of  the  underlying 
facts  or  material  required  to  perform  the  behavioral  objective  of  the 
segment. 

"Help"  refers  to  auxiliary  information  displays  to  aid  student  learning. 
By  this  is  meant  an  auxiliary  display  such  as  a flow  diagram,  a mnemonic, 
an  expanded  generality,  an  algorithm,  or  a decision  tree. 

In  the  "instance  spec,"  a variety  of  instances  of  the  generality  are 
outlined  ("type  description")  with  suggested  "format  description"  for  their 
presentation  or  testing.  The  "CEA"  should  specify  common  logical  errors 
the  student  might  make,  while  the  "MCS"  states  the  minimum  number  (set)  of 
instances  the  student  must  answer  correctly  to  pass,  "instance  production" 
should  state  the  total  number  of  example,  practice,  and  test  items  the 
author  must  produce  for  each  type  of  instance.  "Testing  criteria"  is  the 
passing  criterion,  described  as  usually  "one  MCS  worth  of  test  items 
correct."  No  information  is  given  by  Courseware  as  to  how  the  MCS  and 
testing  criteria  for  the  instance  spec  are  actually  determined. 

As  might  be  surmised  from  this  discussion,  the  specification  document 
for  a single  Courseware  lesson  can  be  quite  lengthy.  The  degree  of  detail 
in  the  objectives  and  generalities  often  is  considerable,  perhaps  as  great 
as  the  NATOPS  description.  Conditions  and  standards  are  given,  and  the 
information  concerning  presentation  and  testing,  at  least  for  cognitive 
type  objectives,  is  extensive. 

Level  of  Effort 

The  preparation  of  lesson  specifications  was  the  culminating  activity 
of  Phase  I efforts  and,  as  might  be  expected,  involved  substantial  amounts 
of  effort.  This  could  be  inferred  from  the  previous  discussion  of  what 
comprised  the  lesson  specifications  for  each  of  the  programs.  Reports  of 
time  devoted  to  this  step  confirm  this  expectation. 
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Only  in  the  Cal  span  E-2C  program  was  it  not  possible  to  identify  time 
devoted  specifically  to  the  preparation  of  lesson  specifications.  In  the 
A-6E  program,  Grumman  reports  a total  of  1,914  of  11,658  professional  man- 
hours (16%)  devoted  to  the  preparation  of  lesson  specifications.  In  Course- 
ware's EA-6B  effort,  1,818  of  its  total  of  6,135  professional  hours  (30%) 
were  devoted  to  this  step,  and  995  of  1,547  Navy  SME  hours  (64%)  were  so 
used.  In  the  Courseware  SH-2F  program,  of  the  contractor's  3,469  professional 
man-hours,  1,305  (38%)  were  devoted  to  lesson  specifications,  while  3,480 
(61%)  of  5,752  Navy  SME  hours  fell  into  this  category. 

Courseware  takes  specific  note  of  the  heavy  manpower  requirement  for  the 
writing  of  lesson  specifications.  In  their  two  programs,  roughly  one-third 
of  the  contractor's  effort  and  two-thirds  of  the  Navy  SME  effort  were  devoted 
to  this  task.  As  can  be  inferred  from  the  previous  discussions  of  the  much 
greater  level  of  detail  that  characterizes  Courseware's  lesson  specifications, 
as  contrasted  with  those  of  Grumnan  and  Cal  span,  the  lesson  specification  is 
a central  part  of  Courseware's  ISO  procedure.  This  is  reflected  in  the  effort 
data.  Perhaps  more  than  any  single  step  of  the  ISO  process  discussed,  this 
one  reflects  a fundamental  difference  in  the  approach  of  the  various  contractors. 

MEASUREMENT  AND  EVALUATION 

In  a strict  sense,  the  methodologies  being  followed  in  the  various  Phase  I 
efforts  did  not  necessarily  require  a systematic  treatment  of  measurement  and 
evaluation  during  Phase  I.  These  topics  could  be  considered  as  a part  of  the 
Phase  II  effort,  at  least  their  development  into  operational  form.  At  the 
same  time,  the  nature  of  the  ISD  process  is  such  as  to  place  great  emphasis 
on  specific  behavioral  statements  of  objectives  with  clear  definition  of  con- 
ditions and  standards.  Further,  there  is  an  explicit  assumption  of  the  hier- 
archical nature  of  learning  or  skill  mastery,  and  the  resultant  training 
program  sequences  are  built  around  this  assumption.  More  specifically,  these 
programs  assume  that  certain  skills  (i.e.,  the  ability  to  perform  given  tasks 
to  specified  standards)  must  be  mastered  before  the  learning  of  skills  higher 
in  the  hierarchy  can  take  place.  There  is  in  each  of  these  programs,  at  least 
to  a considerable  extent,  the  further  hierarchical  assumption  that  the  most 
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effective  and  efficient  instructional  strategy  generally  involves  a repeti- 
tive sequence  of  cognitive  learning,  followed  by  learning/practice  in  a 
training  device  or  simulator,  followed,  finally,  by  learning/practice  in  the 
aircraft. 

In  view  of  these  general  assumptions  concerning  such  relationships,  it 
is  reasonable  that  there  should  be  a considerable  stress  on  techniques  for 
measurement  and  evaluation.  Logically,  management  of  student  progress  through 
the  learning  hierarchies  requires  an  effective  and  efficient  means  of  deter- 
mining whether  the  student  has  achieved  the  various  objectives. 

Measurement  and  evaluation  in  a flight  training  program  occur  at  a 
variety  of  levels  and  for  a variety  of  purposes.  The  internal  and  external 
evaluation  functions  described  in  most  ISO  systems  are  examples  of  this,  as 
is  the  day-to-day  use  of  performance  indices  by  the  instructor  to  prescribe 
instructional  experiences  and  to  guide  the  student  through  the  instructional 
sequence.  It  is  worth  noting,  however,  that  there  are  fundamental  and  prac- 
tical differences  between  the  measurement  of  trainee  performance  with  reference 
to  cognitive  objectives  and  the  measurement  of  trainee  performance  with  ref- 
erence to  complex  psychomotor  objectives  (i.e.,  actual  flight  performance). 

The  technology  for  cognitive  skills  measurement  is  relatively  well  developed 
and  simple  to  implement  in  comparison  with  that  for  flight  skills  measurement. 
Thus,  ability  to  carry  out  the  intended  hierarchical  learning  sequences  in  the 
flight  training  portion  of  the  program,  and  even  the  simulator  training  portion, 
may  be  less  than  that  for  the  academic  portion  by  virtue  of  this  factor. 

All  four  of  the  ISO  programs  exhibit  recognition  of  the  importance  of 
measurement  in  one  way  or  another,  though  none  really  deals  at  length  with 
the  mechanics  of  a measurement  system  in  the  sense  discussed  here.  They  have, 
however,  provided  much  of  the  basic  groundwork  (i.e.,  SBOs,  conditions,  stan- 
dards, etc.)  for  later  development  of  measurement  systems. 

A-6E 

In  the  A-6E  program,  Grumman  provides  SBO  statements  that  contain  stan- 
dards that  vary  in  specificity.  Generally,  the  parameters  that  would  be 
reflected  in  a measurement  system  are  there,  but  no  discussion  of  the  mechanics 
of  measurement  is  given.  There  is  a clear  recognition  of  the  "Initial  Learning— 
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Practice--Oen»/Test"  continuum,  with  a statement  for  each  session  within  a 
lesson  as  to  the  general  manner  In  which  the  objectives  are  to  be  tested. 

For  example,  for  a lesson  within  the  advanced  navigation  unit  there  are 
three  sessions.  In  the  first,  the  student  must  use  a sound/slide  medium 
to  learn  recognition  and  identification  of  navigation  displays  and  symbols. 

The  test  of  this  would  be  a demonstration  in  the  WST.  The  second  session 
might  involve  sound/slide  presentation  of  parameters  and  capabilities  of 
the  system,  with  the  test  being  a written  test.  Session  three  might  require 
flying  a navigation  mission  in  the  WST,  followed  by  a test  navigation  flight 
in  the  aircraft.  While  there  is  a clear  recognition  of  when  and  where  meas- 
urement will  take  place,  there  is  little  detailed  information  concerning 
how.  The  problems  of  flight  versus  non-flight  measurement  are  not  discussed. 

E-2C 

The  recognition  of  measurement  requirements  is  clear  in  the  E-2C  program, 
with  a section  of  the  final  report  devoted  to  testing.  It  states,  "Individual 
performance  evaluation  has  been  addressed  as  a key  component  in  the  contrac- 
tor's E-2C  training  program."  In  fact,  Calspan  developed  a preliminary  data 
bank  for  test  item  development  for  cognitive  objectives.  Similarly,  they 
suggest  the  formulation  of  "pocket  checklists"  from  the  evaluation  criteria 
of  the  behavioral  objectives  "for  the  instructors  to  use  while  assessing 
student  performance  In  practice  and  sortie/scenario  lessons."  Calspan  also 
provides  guidance  as  to  when  and  where  assessment  occurs  in  the  program  for 
each  objective.  A proposed  sequence  of  procedures  for  cognitive  testing, 
involving  pre-testing  and  adaptive  testing,  is  described  in  some  detail,  and 
provides  some  suggestion  of  how.  Other  than  the  "pocket  checklist"  sugges- 
tion, there  is  no  real  discussion  of  the  mechanics  of  flight  skill  measurement. 

EA-6B  and  SH-2F 

In  the  reports  of  these  programs,  particularly  the  EA-6B,  there  is  a 
recognition  of  the  total  system  context  for  ISO  and  the  role  of  testing  in 
that  system.  There  is  discussion  of  the  production  of  tests  and  of  quality 
control  as  specific  steps  in  ISO.  The  testing  information  derives  from  the 
lesson  specification  documents  which,  as  has  been  noted,  are  quite  detailed. 
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Both  the  testing  of  individuals  and  the  internal  and  external  evaluation 
functions  are  noted.  The  lesson  specification  presents  suggested  test  item 
formats  (usually  a sample  item),  the  various  item  types  to  be  represented  in 
that  format,  the  minimum  critical  set  (i.e.,  the  number  of  each  item  type 
that  the  trainee  must  get  correct  to  be  considered  as  passing),  and  the 
number  that  must  be  prepared  of  example,  practice,  and  test  items  (instance 
production).  The  stress  is  almost  completely  on  cognitive  testing,  and 
Courseware  does  present  a great  deal  of  information  in  the  lesson  specifica- 
tion that  is  pertinent  to  cognitive  testing.  Flight  skill  measurement  is 
not  treated  directly. 

Level  of  Effort 

It  was  not  possible  to  determine  the  amount  of  effort  devoted  specifi- 
cally to  the  topic  of  measurement/evaluation  by  the  various  contractors. 

Their  efforts  in  this  direction  were  subsumed  under  the  basic  task  analytic 
activities  and  the  development  of  lesson  specifications. 

DOCUMENTATION 

The  four  programs  produced  a considerable  volume  of  documentation.  The 
general  nature  and  numbers  of  documents  were  determined  by  contractual  re- 
quirements, so  there  was  little  difference  in  the  nuirber  of  deliverable  items 
from  one  program  to  the  other.  As  can  be  surmised  from  the  information  al- 
ready presented,  though,  there  were  some  major  differences  among  the  programs 
in  the  details  of  the  content  and  format  of  the  various  documents  produced. 
While  it  is  not  the  function  of  the  present  report  to  evaluate  the  quality 
of  the  documentation  of  the  three  contractors  in  either  an  absolute  or  a 
comparative  sense,  a few  descriptive  statements  can  be  made  describing  the 
general  nature  of  documentation  for  the  programs. 

The  general  documentation  required  under  the  various  contracts  was  as 
follows: 

• Work  Plan  Report 

• Quarterly  Progress  Reports 

• Task  Listing 
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• Development  of  the  Behavioral  Objectives 

• Training  Support  Requirements 

• Lesson  Specification  Documents 

• Preliminary  Technical  Report 

• Final  Technical  Report 

Not  all  of  the  above  documents  were  available  for  the  present  review  for 
all  programs.  The  bulk  of  this  review  was  based  on  the  final  reports,  lesson 
specifications,  and  behavioral  objectives  documents,  though  even  some  of 
these  documents  were  not  available  for  review. 

The  differences  among  the  programs  with  reference  to  format  and  detail 
in  such  documents  as  task  listings,  behavioral  objectives,  and  lesson  speci- 
fications have  already  been  discussed.  In  general,  the  Courseware  documents 
present  the  greatest  detail  and  Grumman  the  least.  Most  of  the  documents  are 
written  with  reasonable  clarity  and  precision,  though  some  of  the  lesson 
specification  documents  were  in  rough  form  (handwritten)  as  originally  pre- 
pared by  the  SMEs  and  have  likely  benefited  from  subsequent  editing  and 
sharpening. 

Each  of  the  reports  has  areas  in  which  procedures  used  are  not  clearly 
specified  or  are  not  mentioned  at  all.  This  is  to  be  expected  as  a function 
of  the  differences  in  approach  used  and  the  extent  to  which  the  contractor 
has  had  previous  opportunities  to  develop  his  procedures  and  approach. 

Much  of  the  reason  for  Courseware's  having  produced  a greater  volume  and 
detail  of  documentation  results  from  its  having  available  an  already  well 
developed  and  articulated  procedure.  In  addition,  their  use  of  explicit 
algorithmic  approaches  for  many  aspects  of  the  ISD  effort  contributes  to  the 
detail  of  the  documentation. 

Obviously,  much  of  the  effort  that  went  into  the  various  documentation 
items  was  covered  under  the  previously  discussed  topics  such  as  lesson  speci- 
fications. In  terms  of  the  effort  devoted  to  the  documentation  per  se  in 
those  areas,  none  of  the  contractors  reported  it  separately.  With  reference 
to  the  final  report  itself,  only  Grumman  reported  the  level  of  effort  involved. 
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They  have  reported  783  and  696  hours  for  the  instructional  psychologist  and 
educational  specialist,  respectively,  devoted  to  the  final  report.  This 
1,479  hours  represents  about  13  percent  of  the  total  effort  reported  by  Gruman. 

If  Grutnman's  experience  is  at  all  typical,  a fairly  significant  portion 
of  the  total  cost  of  these  Phase  I efforts  was  attributable  to  the  final 
report  documentation.  Considering  the  dual  purpose  of  these  efforts--! .e. , 
both  the  development  of  training  programs  and  the  advancement  of  the  ISO 
technology--this  is  not  surprising. 

GENERAL  PROGRAM  SUMMARIES 

The  previous  discussion  has  presented  the  various  programs  in  a some- 
what fragmented  fashion,  seeking  to  examine  them  with  reference  to  the  major 
functions  performed  in  the  Phase  I ISO  efforts.  It  has  not  been  possible 
to  make  pointed  comparisons  of  the  programs  in  many  areas,  because  the 
information  for  such  comparisons  was  simply  not  available.  Program  pro- 
cedures and  products  have  been  described  and  compared  where  possible,  and 
level  of  effort  devoted  to  the  various  activities  has  been  presented  where 
available.  In  an  effort  to  provide  a somewhat  more  integrated  view  of 
the  level  of  effort,  the  following  tables  present  hours  by  function  as 
given  in  the  final  reports  of  the  various  programs.  These  data  have  gener- 
ally been  presented  previously  in  this  report  in  the  various  sections  dealing 
with  the  ISO  functional  activities  or  steps.  In  examining  these  overall 
program  presentations,  the  reader  must  recognize  that  the  programs  differed 
in  size,  complexity,  number  of  tasks,  number  of  crew  positions,  nature  of 
aircraft  and  systems  involved,  and  in  many  other  ways.  Thus,  it  is  diffi- 
cult to  make  meaningful  evaluative  comparisons  of  the  amount  of  effort 
devoted  to  a given  ISO  function  from  one  program  to  another.  i 

A-6E 

As  noted  previously,  Grumman's  ISO  team  consisted  of  an  instructional 
psychologist  (IP),  educational  specialists  (ES),  and  contractor  and  Navy 
SME  personnel  (SMEc  and  SMEn) . Table  4 shows  the  distribution  of  effort 
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(labor  hours)  of  these  personnel  by  ISO  program  activity.  In  addition,  I 

the  total  time  of  support  personnel  (SP)  reported  for  the  program  is  shown.  | 

Of  considerable  interest  is  Gruirman's  reporting  of  "estimated"  and  "actual"  j 

hours  spent  on  each  of  the  activities.  Table  4 shows  both  sets  of  hours-J-^ 

Grumman  was  the  only  contractor  to  present  estimated  (presumably  the  level  ! 

of  effort  estimates  contained  in  their  proposal)  and  actual  program  level  I 

of  effort  data.  It  is  presumed  that  the  other  contractors'  presentations  j 

of  hours  are  actual  times  devoted  to  the  various  activities.  j 

In  examining  Table  4 several  things  of  interest  can  be  noted.  Most 
obvious  is  the  extent  to  which  the  actual  professional  effort  was  under- 
estimated by  Grumman.  The  ratio  of  total  "actual"  professional  effort  to 
total  "estimated"  is  1.54.  The  task  analysis  step  required  almost  twice  as 
much  effort  as  estimated,  while  the  SBOs  and  final  reporting  took  about  1.5 
times  as  much  as  their  estimates.  The  largest  relative  change  was  for  media 
analysis  which  required  4.0  times  the  estimated  level  of  effort.  Interest- 
ingly, in  the  Grumman  program  only  the  lessor  specifications  step  required 
less  actual  effort  (by  about  10%  percent)  than  was  estimated. 

The  original  Grumman  estimated  levels  of  professional  effort  showed  about 
one-fourth  the  effort  to  be  devoted  to  each  of  the  steps,  task  analysis,  SBOs, 
and  lesson  specifications.  The  actual  effort  shows  almost  one-third  devoted 
to  task  analysis  and  one-sixth  to  lesson  specifications,  with  SBOs  remaining 
at  about  one-fourth  of  the  total . 

imes  shown  in  Table  4 are  given  in  hours,  whereas  they  were 
reported  in  Grumman 's  final  report  in  terms  cf  man-months.  Hour  conver- 
sions were  made  on  the  basis  that  one  man-month  equals  174  hours. 


TABLE  4.  ESTIMATED  AND  ACTUAL  EFFORT  (HOURS)  BY  ISD  FUNCTION:  A-6E  PROGRAM 
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E-2C 

The  detail  shown  by  Cal  span  in  reporting  its  E-2C  effort  by  task  is  not 
as  great  as  that  for  the  A-6E  program  just  discussed.  Only  four  task  cate- 
gories are  distinguished,  and  professional  man-hours  are  not  allocated  by 
contractor  personnel  types.  In  contrast,  however,  Calspan  does  report 
administrative  and  support  man-hours  by  task,  and  times  are  reported  sep- 
arately for  ISO  steps  for  Pilot  position  and  ISO  steps  for  the  NFO/FT 
positions.  Also,  while  Navy  SME  time  is  allocated  by  task,  it  includes 
only  "consultation  time,  not  including  independent  review  of  supporting 
documentation"  by  Navy  personnel.  The  E-2C  program  time  data  are  shown  in 
Table  5. 


TABLE  5.  EFFORT  (HOURS)  BY  ISO  FUNCTION:  E-2C  PROGRAM 


ISO  TASK 

Professional 

Contractor 

Effort 

N^yy 

Admin.  & Support 
Effort 

1. 

Planning 

224 

-0- 

204 

2. 

a.  Pilot  Task  Analysis 

603 

40 

250 

b.  NFO/FT  Task  Analysis 

1,137 

-0- 

447 

3. 

a.  Pilot  Training  Objectives 

603 

10 

130 

b.  NFO/FT  Training  Objectives 

362 

9 

202 

4. 

a.  Pilot  Instructional  Objectives  1,157 

10 

315 

b.  NFO/FT  Instructional 
Objectives 

2,495 

18 

361 

TOTALS 

6,581 

87 

1,909 

While  the  categories  of  ISO  tasks  differ  from  those  used  in  the  Grumman 
listing,  it  is  of  interest  to  note  that  the  proportions  of  contractor  profes- 
sional effort  devoted  to  planning  and  to  task  analysis  are  quite  similar  for 
the  two  programs.  In  each  case,  the  task  analysis  takes  a little  more  than 
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one-fourth  of  the  total  contractor  professional  effort.  Cal  span's  fourth 
category--! nstructional  object! ves--would  seem  to  encompass  Grumman 's  media 
analysis  and  lesson  specification  categories,  and  perhaps  the  final  reporting. 

As  can  be  seen  in  Table  5,  Calspan  devoted  almost  three-fourths  again  as 
many  man-hours  to  the  NFO/FT  ISO  effort  as  to  the  pilot  ISO  effort.  It  will 
be  recalled  that  there  are  three  NFO/FT  crew  positions  involved  (CICO,  ACO, 
and  FT),  so  this  ratio  of  effort  is  understandable. 

In  discussing  the  distribution  of  time  over  the  various  ISO  tasks,  the 
Calspan  final  report  presents  some  important  cautions  concerning  interpre- 
tation. They  note  the  cumulative  nature  of  the  various  tasks,  i.e.,  the 
learning  or  experience  carry  over  from  one  task  to  another,  and  observe  that 
this  makes  it  difficult  to  conclude  from  any  such  data  that  a given  ISO  step 
or  task  will  require  "X"  hours.  They  also  note  that  labor  distribution  is 
affected  by  such  factors  as  prior  availability  of  data,  SME  availability 
within  the  contractor  team  and  from  the  Navy,  computer  modeling  and  avail- 
ability, and  quantity  and  quality  of  an  existing  training  curriculum  and 
supporting  equipment.  These  observations  are  well  worth  bearing  in  mind  with 
reference  to  the  data  contained  in  the  present  report. 

EA-6B 

Courseware  reported  its  effort  in  somewhat  finer  detail  than  did  either 
Grumman  or  Calspan.  The  major  time  category  additions  used  by  Courseware 
include  task  selection,  hierarchy  analysis,  and  sequencing  and  grouping  of 
objectives.  Time  distribution  by  task  for  the  EA-6B  program  is  shown  in 
Table  6.  On  the  EA-6B  effort.  Courseware  reported  a sizeable  number  of  hours 
devoted  to  a variety  of  activities  related  to  the  program,  but  which  are  not 
explicitly  allocated  to  the  ISO  steps.  This  would  include  activities  such 
as  direct  program  technical  management,  internal  reviews,  graphics  support, 
and  the  like.  These  hours  are  shown  in  Table  6 under  the  task  heading  of 
"Other."  The  columns  headed  IP  and  IT  refer  to  Instructional  Psychologist 
and  Instructional  Technologist  personnel,  respectively.  Additional  per- 
sonnel headings  in  the  table  include  Management,  Technical  (mostly  graphics), 
and  Support  (secretarial).  SMEc  and  SMEn  refer  to  contractor  and  Navy  SME 
personnel . 
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TABLE  6.  EFFORT  (HOURS)  BY  ISO  FUNCTION:  EA-6B  PROGRAM 


ISO  TASK 

IP 

IT 

Mqmt. 

Tech. 

Sup't. 

SMEc 

SMEo 

1. 

Job  Analysis 

212 

100 

3 

3 CL 

o -*• 

3 

3 CL 

O -*• 

3 

3 Cl 

O -• 

133 

247 

2. 

Selection  of  Task 

4 

-0- 

<-«■  < 

Ol 

r*‘  < 

Of  CL 

< 

Of  cx 

-0- 

-0- 

3. 

Hierarchy  Analysis 

136 

365 

< C 

0/  Of 

a;  — s 

< C 

Of  Of 
w* 

Of  —4 

< c 

Of  Of 

Of  -H 

235 

208 

4. 

Sequen.  & Grouping 

115 

258 

CT  fiW 
— * 

O ^ 

O’  Of 
— • (/) 
fp  ^ 

O’  Of 

i/t 

289 

97 

5. 

Media  Selection 

152 

-0- 

-H 

O 

rt* 

Of 

o* 

-H 

O 

ST 

-0- 

-0- 

6. 

Lesson  Specification 

459 

752 

(/> 

in 

(/) 

607 

995 

7. 

Other 

1,242 

181 

523 

454 

2,871 

372 

-0- 

TOTALS 

2,320 

1,656 

523 

454 

2,871 

1,636 

1,547 

It  should  be  noted  how  the  "Other"  allocations  were  derived  for  the  various 
personnel  categories  in  Table  6.  Courseware  reported  the  individual  totals 
for  ISO  Tasks  1-6  as  shown  in  the  table,  describing  them  as  "on-site"  per- 
sonnel hours  spent.  In  addition,  they  report  "total  project-related  hours 
spent  by  on-site  contractor  personnel,"  For  the  IT,  for  example,  total  hours 
were  1,656,  whereas  the  IT's  ISO  tasks  amounted  to  only  1,475  hours.  This  181 
hour  difference  was  attributed  to  the  "Other"  category  for  the  IT  in  Table  6. 

In  similar  fashion,  "Other"  totals  were  derived  for  the  SMEc  and  IP  categories. 
In  the  IP's  case,  however,  to  the  difference  of  91  hours  was  added  some  1,151 
additional  hours  reported  by  Courseware  as  project  hours  by  "off-site"  pro- 
fessional personnel,  but  not  otherwise  allocated  by  function. 

As  can  be  seen  from  Table  6,  the  relative  distribution  of  time  by  task 
in  the  EA-6B  effort  differs  from  that  of  the  two  previous  programs.  Most 
notable,  probably,  is  the  lesser  relative  emphasis  on  task  analysis  activities 
(i.e. , the  job  analysis,  task  selection,  and  hierarchy  analysis  tasks)  in  the 
EA-6B  program  and  the  relatively  large  amount  of  time  in  the  "Other"  category. 
Various  possible  reasons  for  this  difference  have  been  discussed  previously, 
but  it  was  not  possible  to  identify  the  actual  causes. 
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SH-2F 

Time  data  for  Courseware's  SH-2F  effort  are  shown  in  Table  7.  The  same 
ISO  task  categories  as  in  the  preceding  table  are  used  here,  except  that 
there  is  no  "Other"  task  category.  There  was  no  separate  reporting  of  on- 
site and  off-site  effort,  nor  were  any  personnel  categories  reported  other 
than  those  shown  in  Table  7. 


TABLE  7.  EFFORT 
SH 

(HOURS)  BY  ISD 
-2F  PROGRAM 

FUNCTION: 

ISD  TASK 

IP 

IT 

SMEn 

1. 

Job  Analysis 

160 

320 

640 

2. 

Selection  of  Task 

40 

40 

80 

3. 

Hierarchy  Analysis 

348 

696 

1,392 

4. 

Sequencing  & Grouping 

80 

160 

160 

5. 

Media  Selection 

80 

240 

-0- 

6. 

Lesson  Specification 

435 

870 

3,480 

TOTALS 

1,143 

2,326 

5,752 

Again,  there  is  reflected  a difference  in  emphasis  from  the  other  programs. 
In  the  SH-2F  program  about  half  of  the  contractor's  effort  went  into  the  task 
analytic  activities  (i.e..  Tasks  1,  2,  and  3).  It  is  extremely  difficult  to 
equate  task  categories  from  one  contractor  to  another,  so  comparisons  are 
tenuous.  It  might  be  assumed,  however,  that  there  is  some  commonality  of 
meaning  of  these  ISD  task  categories  for  the  two  programs  conducted  by  the 
same  contractor.  If  this  is  so,  then  the  difference  in  task  analytic 
requirements  between  the  SH-2F  (46  percent  of  total  contractor  professional 
effort)  and  EA-6B  (19  percent  of  total  contractor  professional  effort)  pro- 
grams is  rather  striking.  However,  the  previous  statements  concerning  the 
multiplicity  of  factors  that  might  relate  to  such  differences  should  be  noted 
even  when  the  same  contractor  is  Involved. 
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Overview 

In  an  effort  to  provide  an  overview  perspective  of  the  manner  in  which 
professional  effort  was  applied  in  the  four  programs,  some  assumptions  will 
be  made  here  concerning  the  essential  equivalency  of  certain  of  the  ISO  task 
categories  across  the  programs  for  descriptive  or  expository  purposes. 

In  reviewing  the  various  categories  of  effort  reported  by  the  contrac- 
tors, a "lowest  common  denominator"  set  of  categories  was  developed.  These 
categories  of  effort  of  ISD  tasks  are  as  follows:  (1)  Work  Plan;  (2)  Task 
Analysis;  (3)  Development  of  Training  Objectives;  (4)  Development  of  Training 
Program  and  Lesson  Specifications;  and  (5)  Other  Activities.  These  cate- 
gories, for  present  purposes,  will  be  labeled  as  "Effort  Categories."  Table  8 
shows  the  manner  in  which  the  various  steps  or  activities  reported  by  the 
contractors  have  been  assigned  to  these  effort  categories. 

Some  of  the  differences  among  the  programs  in  terms  of  what  constituted 
a given  ISO  activity  have  already  been  discussed.  For  example,  one  con- 
tractor may  have  included  activity  within  its  "Task  Analysis"  category  that 
was  not  performed  until  later  by  another  contractor  (e.g..  Courseware's 
hierarchy  analysis).  Another  important  factor  that  should  be  kept  in  mind 
is  the  amount  of  Navy  SME  effort  that  was  available  to  each  of  the  programs 
and  how  it  was  applied.  In  spite  of  these  complications,  the  schema  rep- 
resented in  Table  8 is  of  utility  in  examining  the  four  programs.  In  Table  9 
are  shown  the  numbers  of  contractor  professional  hours  and  the  percentages 
of  total  contractor  professional  effort  they  represent  for  each  of  the  five 
effort  categories.  Navy  SME  total  hours  are  shown  in  the  table  as  an  effort 
category  since  it  was  not  possible  to  distrioute  them  across  effort  cate- 
gories for  all  programs. 

The  data  display  in  Table  9 is  colored  somewhat  by  the  relatively  large 
amount  of  time  in  the  "Other"  category  for  the  A-6E  and  EA-6B  programs.  If 
one  assumed  that  those  hours  were  distributed  over  the  other  four  effort 
categories  in  proportion  to  the  hours  already  in  those  categories  (a  perhaps 
questionable  assumption),  the  apparent  lesser  proportion  of  effort  on  task 
analysis  in  the  EA-6B  program  would  not  be  so  pronounced,  and  conversely  the 
proportion  of  effort  in  category  4 would  increase.  Even  so,  it  is  reasonable 
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to  conclude  that  the  EA-6B  program  devoted  proportionately  less  effort  to 
categories  2 and  3 (task  analysis  and  training  objectives),  and  proportion- 
ately more  effort  to  category  4 (training  program  and  lesson  specification 
development)  than  did  the  other  three  programs.  The  extent  to  which  this 
reflects  a real  systemic  difference  in  approach  or  procedures  is  conjectural. 
One  could  advance  many  reasonable  hypotheses  to  explain  such  an  outcome  other 
than  differences  in  approach  (e.g.,  prior  existence  of  task  data,  nature  of 
aircraft  and  crew  jobs,  etc.).  However,  the  previous  description  of  proce- 
dures used  in  the  various  ISO  steps  for  the  different  programs  suggests  that 
the  EA-6B  effort  was  the  most  highly  procedural i zed  in  its  execution  and  made 
the  greatest  use  of  Navy  SME  resources.  Thus,  the  time  distribution  data  are 
probably  indicative  of  fundamental  systemic  differences  in  the  ISD  approaches 
used  in  the  various  programs.  The  long-term  implications  of  such  differences 
for  training  program  effectiveness  cannot  be  determined  at  this  point,  and 
even  after  completion  of  Phase  II  of  the  various  programs,  it  would  be  ex- 
tremely difficult,  if  not  impossible,  to  identify  the  specific  factors  that 
might  underlie  training  effectiveness  differences. 
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SECTION  III 
COMMENTARY 

INTRODUCTION 

The  previous  section  has  described  the  four  Phase  I SAT/ISD  programs  in 
terms  of  the  various  tasks  or  activities  performed  and  the  general  manner  in 
which  they  were  carried  out.  As  has  been  noted,  the  Navy  had  two  general 
goals  in  embarking  on  these  efforts:  (1)  the  development  of  effective  train- 
ing for  the  four  aircraft  of  concern;  and  (2)  the  advancement  of  the  general 
SAT/ISD  methodology.  The  ultimate  test  of  whether  the  first  of  these  goals 
has  been  achieved  must  await  the  completion  of  the  ISD  process  during  a second 
phase  of  the  various  programs,  i.e.,  the  completion  and  institution  of  the 
instructional  systems  developed,  and  the  evaluation  of  those  instructional 
systems  operationally  in  terms  of  their  cost  and  training  effectiveness.  Only 
the  SH-2F  program  has  progressed  to  Phase  II,  and  even  here  it  will  be  some 
time  before  evaluation  data  eventuate.  At  the  present  time,  any  evaluation  of 
the  Phase  I efforts  must,  necessarily,  be  based  on  their  products,  products  that 
are  intermediate  or  enabling  activities  or  steps  in  the  overall  ISD  process. 

With  reference  to  the  second  goal  area,  that  dealing  with  advancement  of 
the  SAT/ISD  methodology,  there  are  "lessons  learned"  in  these  four  efforts 
that  are  worth  noting.  While  one  may  argue  that  these  lessons  are  somewhat 
limited  by  virtue  of  the  Phase  II  efforts'  not  having  been  completed,  never- 
theless future  Navy  training  development  efforts  can  benefit  from  examination 
of  these  programs.  It  must  be  kept  in  mind,  however,  that  the  four  programs 
were  conducted  in  response  to  the  Navy's  general  conception  of  SAT  method- 
ology as  stated  in  the  early  1975  RFP.  Changes  have  already  taken  place  in 
the  Navy's  conception  of  SAT  and  ISD,  both  as  a result  of  these  four  programs, 
and  as  a result  of  other  SAT/ISD  R&D  efforts.  In  particular,  efforts  being 
conducted  for  the  Navy  by  research  personnel  of  Courseware,  Inc.,  and  Logicon 
have  developed  a very  detailed  model  of  the  SAT/ISD  process  for  specific  use 
in  the  development  of  fleet  aviation  training  programs.  This  model  aims  at 
identifying  all  major  events  or  steps  for  virtually  all  agencies  that  might 
be  associated  with  a given  SAT/ISD  effort.  It  extends  from  the  initial 
identification  of  a potential  requirement  for  a training  development  effort 
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by  fleet  or  other  Navy  personnel,  and  the  identification  of  funds  to  sup- 
port possible  action  on  any  such  development,  to  the  external  quality  control 
evaluation  of  the  implemented  training  program.  The  product  of  this  recent 
Courseware-Logicon  effort  is  much  more  detailed  than  the  procedures  docu- 
mented in  the  four  SAT/ISD  programs  described  here  and,  in  fact,  provides 
guidance  both  for  the  conduct  of  an  ISD  effort  and  for  its  procurement  and 
management  by  the  Navy.  At  the  same  time,  the  model  is  fundamentally  the 
same  procedure  used  by  Courseware  in  its  EA-6B  and  SH-2F  programs  discussed 
in  this  report. 

In  this  commentary  section  of  the  report,  an  effort  is  made  to  examine  all 
of  these  various  efforts  in  terms  of  their  implications  for  improving  future 
Navy  aviation  training  development  efforts.  The  framework  for  the  coimients 
made  is  necessarily  subjective  and  is  based  on  the  experiences  of  the  author 
and  his  professional  colleagues  over  a number  of  years  of  aviation  training 
and  simulation  research  and  development  activities.  It  also  reflects,  perhaps 
in  a selective  fashion,  the  considerable  body  of  literature  dealing  with  the 
systems  approach  to  training  and/or  with  instructional  systems  development. 

Goals  of  ISD 

In  any  such  review  of  programs  or  methodologies  as  the  present  one,  it  is 
appropriate  to  have  some  idea  of  the  general  goals  of  those  programs  or  metho- 
ologies.  In  the  case  of  ISD,  the  general  goal  is  the  development  of  more 
cost  effective  training  programs  than  would  be  the  case  without  ISD.  In 
aviation  training,  in  particular,  the  desire  for  cost  effective  training  is  of 
special  concern  to  DoD  because  of  the  critical  part  aviation  forces  play  in 
our  national  defense  and  because  of  the  high  levels  of  cost  associated  with 
aviation  training.  Aviation  training  is  one  of  the  most  costly  types  of 
military  training  because  of:  (1)  the  difficulty  of  the  skills  involved  and 
the  consequent  years  of  training  required  to  develop  combat  mission-capable 
personnel;  (2)  the  extremely  high  quality  of  the  personnel  resources  required 
to  operate  and  maintain  aircraft;  and  (3)  the  high  hourly  operating  costs  of 
the  increasingly  complex  aircraft  that  comprise  the  current  inventory.  Add 
to  these  factors  concern  over  availability  of  fuel  and  concern  for  the  en- 
vironment, and  it  is  not  surprising  that  development  of  means  of  reducing 


NAVTRAEQUIPlEN  77-C-0009-1 


aviation  training  costs  while  maintaining  necessary  combat  readiness  is  one 
of  the  most  critical  problem  areas  among  all  the  services.  ISD  has  been 
pursued  because  of  its  seeming  potential  for  maintaining  or  improving  the 
quality  of  training  at  least  possible  cost.  It  is  with  reference  to  this 
general  cost  effectiveness  goal  or  potential  that  the  comments  here  are 
presented. 

THE  FOUR  SAT/ I SO  PROGRAMS 

The  principal  question  of  concern  with  reference  to  the  four  specific 
SAT/ISD  programs  examined  (i.e.,  the  A-6E,  E-2C,  EA-6B,  and  SH-2F)  is  the 
extent  to  which  they  have  moved  toward  the  goal  of  more  cost  effective 
training.  Clearly,  all  four  represent  systematic  approaches  of  some  thor- 
oughness. For  this  reason,  if  no  other,  they  would  likely  result  in  programs 
that  are  more  effective  than  those  existing  for  the  four  aircraft,  assuming 
that  Phase  II  efforts  are  carried  out.  However,  this  question  warrants 
further  examination. 

Phase  I Adequacy  for  Phase  II 

Each  of  the  programs  has  produced  Phase  I products  that  would,  at  least 
in  theory,  allow  the  conduct  of  Phase  II.  The  documentation  of  the  EA-6B 
and  SH-2F  programs  contains  the  most  detail  and  is  more  than  adequate  to  allow 
authoring  of  actual  lesson  materials,  at  least  for  cognitive  skill  areas. 

Very  little  information  transmission  loss  would  occur  between  Phases  I and  II 
of  these  programs  due  to  the  breadth  and  depth  of  the  documentation.  It 
should  be  noted,  though,  that  less  than  half  the  required  lesson  specifica- 
tions were  produced  in  the  EA-6B  program  and  that  many  of  those  for  the  SH-2F 
were  in  need  of  much  further  development.  Therefore,  when  it  is  concluded 
here  that  Phase  I products  of  these  two  programs  are  more  than  adequate  to 
most  Phase  II  needs,  a qualifying  assumption  concerning  their  completion  is 
required. 

The  documentation  for  the  E-2C  program  also  seems  complete  and  adequate 
for  most  Phase  II  needs.  It  is  somewhat  less  detailed  than  that  of  the 
EA-6B  and  SH-2F  programs,  and  it  would  likely  require  somewhat  more  effort 
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to  author  the  Phase  II  materials,  but  the  essential  information  is  there. 

The  flow  of  instructional  events  in  the  program  that  would  be  developed  in 
Phase  II  is  clear  and  easy  to  follow  in  the  Phase  I materials. 

The  A-6E  program  documentation  would  likely  be  the  most  difficult  to  use 
in  Phase  II.  While  it  gives  a clear  exposition  of  the  objectives  to  be 
covered  in  the  various  units  and  lessons,  it  provides  relatively  little 
information  concerning  details  of  the  lesson  materials  that  would  aid  in 
future  authoring.  It  is  likely  that  the  Phase  I products  are  adequate  for 
the  carrying  out  of  Phase  II,  but  Phase  II  personnel  would  have  to  consult 
secondary  references  (which  are  cited  in  the  lesson  specifications)  ex- 
tensively in  order  to  transform  the  Phase  I program  concept  into  an  opera- 
tional form. 

The  major  common  shortcoming  of  these  four  Phase  I efforts,  in  terns 
of  their  adequacy  for  Phase  II,  is  their  relative  degree  of  orientation 
toward  cognitive  skills,  as  opposed  to  orientation  toward  actual  in-cockpit 
flying  skills.  None  of  the  programs  really  addresses  how  the  training  will 
be  conducted  in  the  cockpit  of  the  aircraft  or  the  simulator.  The  emphasis 
is  more  on  the  cognitive  aspects  of  flying  than  it  is  on  what  to  do,  what 
to  look  for,  how  to  instruct,  and  how  to  evaluate  in  the  cockpit.  Thus,  to 
the  extent  that  achieving  the  ultimate  ISD  goal  of  increased  cost  effec- 
tiveness in  training  is  dependent  on  these  "in-the-cockpit"  factors,  these 
Phase  I efforts  are  less  than  optimal  in  their  utility  for  Phase  II. 

Perhaps  the  second  major  common  shortcoming  is  in  the  area  of  performance 
measurement  and  evaluation.  The  Phase  I products  do  not  adequately  address 
this  area  in  the  manner  required  for  development  in  Phase  II  of  a measure- 
ment system  insofar  as  flight  skills  are  concerned.  The  criticality  of  the 
measurement  area  to  any  training  program,  but  especially  to  an  individualized 
program  based  on  assumed  logical  hierarchies  of  objectives,  has  been  men- 
tioned. While  the  means  for  meeting  necessary  cognitive  skill  measurement 
requirements  could  probably  be  developed  reasonably  well  in  Phase  II  from 
the  Phase  I information,  little  is  said  in  the  Phase  I documents  concerning 
the  problems  of  in-the-cockpit  measurement  of  complex  perceptual -motor 
flight  skills.  Therefore,  developing  an  effective  flight  skill  measurement 
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system  during  Phase  II  would  require  a considerable  effort.  The  specifi- 
cation of  standards  of  fliq.U  performance  provided  In  the  Phase  I documen- 
tation would  be  useful  in  this  regard,  but  the  question  of  a flight  per- 
formance measurement  system  concept  is  not  addressed  at  all.  This  is  a 
matter  of  some  import. 

Skills  and  Cost  Effectiveness 

As  has  been  stated,  aviation  training  managers  are  properly  concerned 
with  cost  effective  training.  The  basic  determining  factors  in  the  high 
cost  of  flight  training  are  the  number  of  aircraft  hours  flown  per  trainee 
and  the  cost  per  flight  instructional  hour.  For  aircraft  of  the  complexity 
of  the  four  examined  here,  these  hourly  flight  instructional  costs  can  be 
$1,000  or  more.  Thus,  it  is  imperative  that  the  number  of  flying  hours  in 
the  program  be  held  to  the  minimum  necessary  for  each  trainee  to  attain 
required  skill  levels  and,  correspondingly,  tiat  each  cockpit  hour  yield  as 
much  instructional  benefit  (i.e.,  training  effectiveness)  as  possible.  This 
should  be  the  goal  of  the  instructional  system  developer,  but  it  is  imperative 
that  he  not  compromise  the  attainment  of  necessary  proficiency  levels  in 
seeking  to  reduce  costs. 

It  follows  that  the  flying  program  must  receive  primary  attention  in  an 
I5D  effort  if  cost  effectiveness  is  to  be  maximized.  It  is  for  this  reason 
that  the  relative  lack  of  attention  to  in-the-cockpit  instructional  exper- 
iences in  the  four  Phase  I efforts  is  of  concern. 

Numerous  research  efforts  over  the  past  decade  have  demonstrated  that  the 
single  greatest  impact  on  flight  training  program  cost  effectiveness  has  been 
achieved  through  effective  use  of  flight  simulation.  The  flight  simulator, 
when  combined  with  an  effective  synthetic  training  program,  has  allowed 
significant  reductions  in  required  flight  hours,  in  many  cases  virtually  to 
the  point  of  eliminating  the  requirement  for  aircraft  time.  Therefore,  when 
a state-of-the-art  flight  simulator  is  available  to  the  ISO  team,  it  should 
be  the  basic  medium  around  which  the  course  is  organized.  Only  in  this  way 
will  maximum  cost  effectiveness  be  realized  in  a flight  training  program. 

It  is  noted,  too,  that  those  programs  that  have  shown  the  most  significant 
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benefits  from  the  use  of  simulation  nave  been  based  on  proficiency-pacino  nrin- 
ciples.  The  concept  of  "train  to  proficiency"  is  well  established  in  terms  of 
its  advantages.  For  an  ISO  effort  truly  to  build  on  this  strategy  or  approach, 
the  focus  must  be  on  in-cockpit  flight  skills,  and  flight  performance  measure- 
ment must  be  treated  effectively  as  a critical  and  integral  part  of  the  over- 
all instructional  system. 

Stated  differently,  concentrating  emphasis  on  in-the-cockpit  flight  skills, 
as  opposed  to  emphasis  on  cognitive  skills,  is  necessary  to  significant  in- 
creases in  cost  effectiveness  in  aviation  training.  This  is  not  meant  to 
imply  that  cognitive  skills  are  not  of  importance,  but  merely  that  an  ISD 
effort  must  concentrate  attention  on  the  most  critical  area,  the  flight  skills 
area,  if  cost  effectiveness  is  to  be  maximized,  and  the  program  should  make 
maximum  use  of  simulation  commensurate  with  cost  effectiveness  considerations. 

As  was  noted  in  the  previous  section,  the  four  ISO  efforts  examined  de- 
vote much  more  attention  to  the  cognitive  area  than  to  the  flight  skills  area 
(both  aircraft  and  simulator).  Therefore,  their  Phase  II  efforts  may  not  achieve 
the  degree  of  cost  effectiveness  desired  and  possible,  though  all  would  likely 
show  significant  gains  over  present  programs  if  carried  through  Phase  II. 

Another  aspect  of  this  facet  of  the  problem  is  worth  noting.  The  ISD 
procedures  used  in  each  of  the  four  programs  were  fundamentally  very  similar. 

All  assumed  an  hierarchical  structure  of  skills  and  their  learning  to  some 
degree  and,  as  a consequence,  tended  to  an  iterative  instructional  sequence 
of  "cognitive-then  trainer-then  aircraft."  It  is  logical  that  one  should  learn 
the  cognitive  enabling  skills  first  in  the  classroom  (or  in  a carrel,  or  from 
a programmed  text,  etc.)  before  those  skills  are  used  in  the  cockpit.  However, 
it  has  been  our  experience  that  many  such  "enabling"  items  can  best  be  learned 
directly  in  the  cockpit  context  when  and  as  they  are  needed  in  the  flight 
mission  performance.  Thus,  in  some  programs  the  classroom  has  been  virtually 
eliminated  as  the  locus  of  instruction  in  favor  of  the  procedures  trainer  or 
simulator.  Not  only  does  this  provide  a true  functional  context  for  the  in- 
struction, with  attendant  benefits  to  both  learning  and  retention,  it  results 
in  the  elimination  of  much  material  that  was  previously  felt  to  be  essential 
based  on  usual  assumed  hierarchial  relationships.  As  a result,  such  programs 
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have  much  smaller  and  simpler  media  requirements. 

The  suggestion  that  this  functional  context  approach  changes  the  nature 
of  required  enabling  objectives  may  seem  strange.  However,  building  a 
hierarchical  structure  to  support  a cognitive  analogue  of  the  cockpit  tasks 
can  be  conceived  as  a different  process  from  building  a hierarchical  struc- 
ture in  the  cockpit.  The  cockpit  context  itself  provides  a cue-response 
support  directly  that  must  be  synthesized  if  the  structure  is  completely 
cognitive.  Because  of  the  fundamental  implications  these  points  have  for 
the  design  and  operation  of  flight  training  courses,  this  is  an  area  that 
warrants  investigation.  The  implications  for  cost  effectiveness  are  con- 
siderable. 

THE  ISO  MODEL 

The  basic  purpose  of  this  report  is  to  identify  areas  in  which  change 
may  improve  Navy  SAT/ISD  programs  in  the  future.  As  noted,  significant 
changes  and  improvements  in  the  ISO  process  conceptualization  have  been 
made  over  the  past  three  years  as  a result  of  a variety  of  Navy  and  non-Navy 
efforts.  The  emerging  Navy  ISD  process  model,  along  with  its  implementing 
procedures  and  instructions,  is  a well -formulated,  well-stated,  and  complete 
approach.  Much  serious  thought  and  effort  have  gone  into  its  development. 

It  offers  promise  for  significant  improvements  in  Naval  aviation  training 
programs  and  for  the  production  of  predictable  results.  Further,  it  pro- 
vides a means  for  more  effective  management  of  ISD  programs. 

For  the  first  time,  perhaps,  the  full  cast  of  agencies  and  personnel  in 
the  ISD  process  is  described  in  the  model.  If  ISD  is  to  work  to  maximum 
effectiveness,  it  requires  the  working  together  of  numerous  agencies  from  the 
CNO  on  down  to  the  training  squadron  and  the  contractor(s)  involved.  The 
conception  is  that  of  an  effort  in  which  many  agencies  must  work  together  to 
produce  the  final  product.  Therefore,  responsibilities  and  authority  must 
be  identified;  the  various  steps  in  the  process  must  be  carefully  described 
and  their  inputs  and  outputs  must  be  stated;  the  nature  of  the  process  must 
be  specified;  and  the  framework  for  managing  and  evaluating  the  effort  must 
be  made  clear.  The  model  does  these  things  explicitly. 
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What,  then,  are  some  of  the  directions  the  Navy  might  take  to  improve 
application  of  the  model  to  futu'^e  training  programs?  In  seeking  to  respond 
to  such  a question,  the  present  effort  is  reacting  to  several  "models"  or 
sets  of  ISO  procedures  as  these  were  manifest  in  the  four  ISO  programs  ex- 
amined and  in  the  subsequent  model  development  by  Courseware  and  Logicon. 

So,  in  this  context  the  areas  for  possible  future  improvement  are  based  on 
a conglomerate  view  of  all  these  efforts.  To  refer  on  this  basis  to  the 
model  may  be  presumptious , but  the  general  direction  of  SAT/ISD  evolution  in 
the  Navy  is  clearly  discernable  from  the  1975  SAT/RFP,  the  four  SAT  programs 
with  their  considerable  documentation,  and  more  recent  Courseware-Logicon 
model  development  (along  with  an  ISO  specification),  even  though  these  latter 
documents  may  not  be  yet  considered  as  official  Navy  position. 

Future  Improvemen ^ 

The  following  suggested  areas  for  future  improvement  of  the  model  and  Navy 
SAT/ISD  programs  can  be  ascertained,  in  part,  from  the  various  observations 
about  the  four  ISD  programs  made  in  the  preceding  Results  section.  Others 
derive  from  evaluation  of  other  documents  and  the  general  direction  of  SAT/ 

ISD  evolution  previously  mentioned.  No  attempt  will  be  made  to  identify  all 
the  sources  from  which  these  suggestions  derive  due  to  the  magnitude  of  the 
material  reviewed  and  its  areas  of  redundancy  and  uniqueness. 

Personnel . It  is  clear  from  the  reports  of  the  four  ISD  programs  and  from 
the  subsequent  articulation  of  the  ISD  model  and  specification  that  all  parties 
recognize  the  criticality  of  pe’^sonnel  to  effective  ISD.  It  is  a labor-inten- 
sive process,  and  a variety  of  types  of  personnel  and  skills  are  required  for 
its  successful  execution.  Two  of  the  principal  types  of  personnel  required 
are  those  knowledgable  In  the  areas  of  ISD  and  instructional  psychology  and 
those  knowledgable  in  the  operational  job  area  of  concern,  i.e.,  the  SME. 

The  model  should  specify  the  required  nature  of  qualifying  training  and  ex-  i 
perience  for  such  personnel. 

With  reference  to  the  SME,  the  model  needs  some  further  definition  of 
their  characteristics  and  qualifications  for  the  various  functional  roles 
they  may  fulfill.  For  example,  execution  of  a task  selection  algorithm  might 
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require  input  from  a variety  of  SMEs.  A fleet  maintenance  SME  who  has  had 
no  contact  with  maintenance  training  would  likely  be  in  no  position  to  eval- 
uate trainee  entry  level  behavior,  while  a fleet  maintenance  supervisor  SME 
might  not  be  able  to  provide  valid  data  concerning  the  frequency  with  which 
a new  fleet  aviation  mechanic  will  perform  a given  task. 

With  reference  to  the  other  personnel  type,  the  ISD/psychology  "expert," 
the  model  is  moving  toward  a detailing  of  required  training  and/or  experience. 
While  it  is  desirable  that  unqualified  instructional  design  personnel  not  be 
utilized,  it  would  appear  unwise  to  move  strongly  in  the  direction  of  rigidly 
specifying  the  necessary  academic  qualifications  of  professional  personnel. 
Effective  knowledge  of  human  learning  and  of  aviation  training  is  not  con- 
fined just  to  those  with  the  Ph.D.  or  Master's  degree  in  instructional  psy- 
chology. It  is  suggested  here  that  the  quality  of  past  training  development 
products  of  the  professional— i .e. , his  demonstrated  capability,  his  inno- 
vativeness, the  cost  effectiveness  of  his  programs,  etc. --is  much  more  critical 
than  his  academic  qualifications.  To  be  unduly  restrictive  in  statement  of 
professional  personnel  qualifications  not  only  will  reduce  the  numbers  of 
personnel  available  to  work  on  Navy  programs,  it  will  not  necessarily  increase 
the  quality.  Many,  or  perhaps  most,  of  the  personnel  who  have  pioneered  ISO 
and  have  been  effective  practitioners  would  not  fit  the  mold  of  the  instruc- 
tional psychologist  that  seems  to  be  evolving  in  the  Navy  ISO  model. 

Role  of  the  SME.  There  is  no  question  as  to  the  importance  of  the  SME's 
role  in  the  ISD  process.  His  input  is  a necessity.  The  evolving  model  recog- 
nizes this,  and  it  also  recognizes  the  need  for  SME  training  for  his  ISD  role. 
The  SME  role  is  particularly  influential  in  defining  and  selecting  tasks  to 
be  trained  and  in  specifying  task  interdependencies  and  sequential  relation- 
ships. As  previously  suggested,  the  SME  has  a variety  of  roles  to  fulfill. 
Review  of  the  four  SAT/ISD  programs  suggests  that  SME  input  was  accepted  in 
a somewhat  uncritical  fashion.  The  model  would  benefit  if  it  provided  more 
clearly  for  a "challenge  and  response"  to  SME  input  by  the  instructional 
developer.  In  particular,  the  need  for  each  of  the  enabling  objectives  needs 
to  be  challenged,  as  does  the  nature  of  the  task  and  objectives  hierarchies. 
This  is  not  to  suggest  that  the  SME  input  is  invalid,  but  that  means  are 
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required  to  Insure  that  each  aspect  of  the  resultant  program  is  functional 
and  necessary. 

Examples  of  SME  input  in  need  of  challenge  include  statements  of  trainee 
input  skill  levels;  statements  of  what  is  and  should  be  taugnt  in  UPT,  at  the 
RTS,  and  in  the  fleet;  and  statements  as  to  standards  of  performance  that 
should  be  required  during  and  at  the  end  of  readiness  squadron  training. 

Areas  such  as  these,  and  particularly  the  enabling  objectives  content  of  the 
program,  have  a marked  influence  on  the  ultimate  nature  and  cost  of  the  pro- 
gram. Therefore,  such  inputs  must  be  subjected  to  careful  scrutiny  and 
validation.  The  model  should  highlight  this  validation  process.  In  fact, 
the  model  should  emphasize  "challenge  and  response"  with  reference  to  all 
activities.  One  of  the  real  strengths  of  the  ISO  procedure  is  the  iterative 
nature  of  its  analysis,  justi^'ication,  and  revision  activities.  Without  the 
challenge  in  the  total  system  framework,  the  ISO  process  will  be  reduced  to 
stereotypy  and  will  largely  lose  its  effectiveness. 

System  Constraints.  Closely  related  to  the  challenge  and  justification 
idea  is  the  handling  in  the  model  of  system  constraints.  Clearly,  the  system 
approach  requires  that  the  instructional  designer  recognize  the  realities  of 
funding  and  resource  constraints  in  the  design  process.  While  the  ultimate 
program  must  be  "real  world"  and  live  within  necessary  constraints,  it  should 
not  be  based  on  an  overly  submissive  acceptance  of  constraints.  Who  is  to  be 
the  advocate  for  new  equipment,  better  simulators,  more  flying  hours,  etc.,  if 
it  is  not  the  ISO  team?  Resources  should  flow  from  requirements,  not  vice 
versa.  For  example,  the  draft  ISO  specification  suggests  that  the  maximum 
number  of  flights  to  be  contained  in  the  syllabus  will  be  determined  by  con- 
straints identified  in  the  problem  analysis,  rather  than  by  the  nature  and 
extent  of  the  training  requirements.  While  such  fixed  factors  may  ultimately 
be  the  deciding  ones,  they  should  not  be  the  starting  point  for  a syllabus 
or  program. 

Documentation.  Review  of  the  proposed  ISD  procedures  reveals  a large 
number  of  contract  deliverable  documents.  Specification  of  deliverable  items 
is  necessary  to  effective  management  of  ISD  programs  by  the  Navy.  Documents 
should  serve  a functional  purpose  in  an  ISD  effort,  so  the  purposes  and  users 
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of  the  various  documents  must  be  identified.  The  mode!  generally  does  a 
thorough  job  in  this  regard.  The  point  that  warrants  future  examination  is 
the  required  number  and  types  of  formal  deliverable  documents  for  Navy  man- 
agement purposes,  and  determination  of  items  that  must  be  prepared  as  inter- 
mediate products,  but  which  do  not  warrant  formal  document  status.  Prepara- 
tion of  a document  as  a contract  deliverable  item  takes  time  and  effort  and, 
hence,  costs  money.  If  a "working  paper"  format  is  sufficient  in  some  in- 
stances for  the  purposes  to  be  served,  time,  effort,  and  money  will  be  saved. 
This  aspect  of  the  model  and  specification  warrants  future  study  and  re- 
fi  nement . 

Another  aspect  of  documentation  relates  to  the  need  for  the  degree  of 
detail  and  graphic  illustration  of  such  items  as  objectives  hierarchies, 
lesson  maps,  and  lesson  specifications.  A given  ISO  practitioner  may  view 
this  detail  as  necessary  to  downstream  work  in  Phase  II  efforts,  whereas 
another  practitioner  might  find  it  less  useful.  It  is  an  empirical  question 
as  to  the  utility  of  the  types  of  information  produced  in  the  various  Phase  I 
efforts  examined  here.  The  question  is  not  whether  Contractor  A would  find 
the  Phase  I information  lie  generates  to  be  useful  in  Phase  II.  Rather,  it  is 
whether  Contractor  B would  be  able  to  use  effectively  in  Phase  II  the  Phase  I 
information  generated  by  A.  In  the  only  systematic  examination^of  the 
utility  of  various  ISO  products  of  which  the  present  author  is  aware,  it  was 
found  that  in  the  Army's  systems  approach  to  training  (a  procedure  not  funda- 
mentally different  from  current  ISO  models)  many  documentary  products  were 
produced  for  which  there  was  no  real  need  or  use.  A more  recent  investiga- 
tion^within  the  USAF  Tactical  Air  Conrmand  of  its  ISO  efforts  also  suggests 
that  documentation  is  a continuing  problem  for  ISO  teams.  The  question  of 
the  utility  of  various  documents  to  the  ISO  process  should  be  investigated 
further,  and  the  results  of  such  investigation  can  be  integrated  into  the 
Navy  ISO  model  to  improve  it  in  the  future. 
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ISP  and  Flight  Skills.  As  has  been  noted  in  earlier  observations,  the 
in-cockpit  flight  skills  must  be  the  focal  point  of  an  ISO  effort  for  pilot 
training.  Much  of  the  procedure  involved  in  most  ISP  models,  including  the 
evolving  Navy  model,  has  derived  from  instructional  psychology  research 
largely  related  to  cognitive  skills.  The  ISP  procedures  have  not  been  as 
well  developed  or  articulated  for  in-cockpit  skills  learning.  As  a result, 
much  remains  to  be  done  to  develop  such  procedures  for  the  in-cockpit  training 
situation.  The  logic  of  ISP  methods  seems  appealingly  applicable  to  the  de- 
velopment of  more  effective  training,  whatever  the  type  of  skill  involved,  but, 
in  fact,  as  a technology,  ISP  is  not  as  well  developed  for  the  cockpit  appli- 
cation as  it  is  for  the  non-cockpit  aspects  of  flying.  The  point  here  is  not 
that  ISP  is  inappropriate  to  the  cockpit,  but  that  further  research  and  de- 
velopment is  necessary  to  make  't  as  effective  for  the  cockpit  setting  as  it 
has  been  for  the  non-cockpit  setting. 

Simulation.  More  than  any  other  single  development,  the  modern  flight 
simulator  has  favorably  affected  both  the  effectiveness  and  efficiency  of 
flight  training  programs.  However,  to  use  the  simulator  to  greatest  cost 
effectiveness  requires  more  than  treating  it  as  just  another  training  medium 
or  treating  it  essentially  the  name  as  an  aircraft.  The  question  is  how  to 
maximize  the  training  capabilities  of  the  simulator  in  a given  training  setting, 
i.e.,  how  to  use  the  simulator  to  maximum  effect  as  a learning  environment. 
Simulation  has  received  a great  deal  of  attention  in  aviation  training,  and 
the  evidence  is  overwhelming  that  the  manner  in  which  the  simulator  is  used 
is  crucial  to  cost  effectiveness  outcomes.  Because  of  such  factors,  and  be- 
cause of  the  centrality  of  simulatioi  to  flight  training,  the  ISP  model  and 
specification  should  highlight  simulation  and  seek  to  insure  that  ISP  efforts 
treat  simulation  appropriately.  It  is  contended  here  that  the  four  programs 
examined  did  not  use  simulation  to  maximum  effect,  though  all  did  rely  on  it 
extensively.  The  model  would  benefit  by  a better  articulation  of  how  to  use 
simulation  to  maximum  effect. 

The  comnents  previously  made  to  the  effect  that  ISP  researchers  have  not 
developed  their  technology  as  well  for  application  to  cockpit  skills  as  to 
cognitive  skills  are  pertinent  here.  The  simulator  presents  a whole-task 
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environment  in  which  complex  perceptual -motor  skills  can  be  presented,  i 

practiced,  and  tested  with  real-time  decision  making  and  temporal  Integra-  ] 

tion  demands  present.  The  "how-to  secrets"  of  doing  this  most  effectively  ] 

have  not  yet  been  reduced  to  a prescriptive  procedure  for  routine  applica-  i 

tion.  As  noted,  the  state-of-the-art  here  is  much  the  same  as  that  for 
aircraft  training  development. 

All  of  this  is  not  to  say  that  nothing  is  known  about  effective  simu- 
lation (or  aircraft)  training.  A great  deal  is  known,  but  it  is  not  as  pro- 
cedural ly  documented  as  what  is  known  about  effective  non-cockpit  training, 
nor  is  it  as  easily  prescriptive.  While  we  have  dwelt  somewhat  on  the 
similarity  of  the  state  of  affairs  with  reference  to  aircraft  and  simulator 
training,  this  is  not  meant  to  suggest  that  the  simulator  should  be  used  in 
the  same  manner  as  the  aircraft.  There  are  differences  between  the  two  media 
that  must  be  recognized  and  used  positively  in  training  program  design. 

The  ISD  model  provides  for  simulation  and  recognizes  it  as  a highly  im- 
portant aspect  of  flight  training.  It  would  be  better  if  it  could  provide 
the  full  recognition  and  detailed  guidance  concerning  effective  use  of  simu- 
lation that  the  subject  warrants.  This  reflects  not  an  oversight  of  the 
ISD  model,  but  a deficiency  in  our  state  of  knowledge  concerning  simulation, 
or,  at  least,  the  adequacy  with  which  that  knowledge  has  been  documented. 

Thus,  improvement  of  the  ISD  model  in  this  area  is  dependent  upon  the  accom- 
plishment of  the  R&D  activities  necessary  to  development  and  statement  of 
the  required  procedural  guidance. 

Measurement.  The  technology  of  flight  performance  measurement  has  re- 
ceived much  less  attention  over  the  years  than  it  warrants.  The  needs  in 
this  area  are  becoming  even  more  evident  as  the  analytical  approaches  of  SAT/  I 

ISD  are  applied  on  an  ever  wider  scale.  The  emphasis  in  these  approaches 
on  careful  statement  of  desired  behavioral  outcomes,  with  specified  conditions 
and  standards  statements,  highlights  these  rreasurement/evaluation  deficien-  ' 1 

cies.  ISD  will  do  less  than  the  desired  good  if  the  student's  progress 
through  the  carefully  engineered  task-learning  sequences  cannot  be  managed 
properly  because  of  inadequate  measurement.  This  is  clearly  an  area  in 
which  technology  base  advancement  is  required  and  in  which  the  ISD  model 
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can  be  improved.  While  the  general  need  for  measurement  is  recognized  in 
the  model,  the  difficulty  of  its  effective  accomplishment  is  not,  nor  arc 
the  differences  between  flight  and  non-flight  measurement. 

Another  aspect  of  this  question  is  also  of  interest  for  future  model 
advancement.  The  general  concept  or  strategy  of  training  that  most  ISD 
programs  support,  at  least  theoretically,  is  that  of  an  individualized, 
proficiency-paced  program.  Research  studies  have  shown  individual  student 
flight  time  variations  of  100%  or  more  when  this  strategy  is  effectively 
used.  The  emerging  Navy  ISO  model  would  permit  such  individualization,  but 
it  does  not  emphasize  it.  To  do  so  assumes  the  measurement  technology 
necessary  to  effective  management  and  control  of  individualized  training. 
Individualization  also  makes  considerable  use  of  diagnostic  testing  se- 
quences for  development  of  individual  training  prescript’cns.  Thus,  there 
are  a variety  of  reasons  why  measurement  is  a critical  need  for  ISD-based 
programs,  reasons  that  go  far  beyond  the  usual  concern  for  qualification 
standards  for  the  individual  student.  Measurement  warrants  more  attention 
in  the  model  and  in  technology  base  programs. 

SUMMARY  COMMENT 

The  Navy  has  moved  forward  significantly  with  the  programs  discussed  in 
this  report.  Previous  SAT/ISD  model  developments  have  been  aimed  at  more 
generalized  applications,  whereas  the  series  of  programs  discussed  here  have 
been  aimed  at  a much  more  restricted  application,  aviation  training,  and  have 
sought  to  target  more  clearly  the  specifics  of  effective  instructional  system 
development  in  aviation.  One  problem  with  the  more  general  models  in  appli- 
cation is  that  each  area  of  application  presents  some  unique  problems  that  may 
render  the  general  model  less  useful  than  desired.  While  aviation  has  many 
commonalities  with  other  training  content  areas,  it  has  many  critical  uniquities 
as  well.  Because  of  this  fact  and  because  of  the  high  cost  and  importance 
of  aviation  training,  the  development  of  an  aviation-specific  ISD  model  is 
justified. 

The  ISO  model  envisioned  by  the  Navy  would  cover  both  operator  and  main- 
tenance personnel  for  fleet  aviation  systems.  Consideration  of  the  unique 
aspects  of  the  flight  skill  acquisition  process,  as  discussed  above,  might 
suggest  that  the  inclusion  of  maintenance  skills  within  the  same  model/ 
specification  as  covers  pilot/aircrew  operator  skills  will  continue  the  types 
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of  application  problems  cited  for  the  more  general  models.  This  question 
warrants  further  consideration  by  the  Navy. 

In  this  examination  of  specific  SAT/ISD  efforts  and  of  the  emerging  ISD 
model,  several  areas  for  future  research  and  improvement  of  the  model  have 
been  identified.  The  most  important  of  these  are:  (1)  the  measurement/ 
evaluation  area;  (2)  the  cognitive  vs.  flight  skill  questions;  and  (3)  the 
role  of  simulation.  These  three  areas,  in  particular,  are  of  high  import 
with  reference  to  the  cost  effectiveness  of  ISD  and,  as  such,  represent 
areas  of  major  potential  gain  in  model  improvement. 

A further  word  is  in  order  concerning  the  last  of  the  three  areas  men- 
tioned, simulation.  In  examining  various  pilot  training  ISD  efforts  conduc- 
ted by  a number  of  different  agencies,  both  government  and  contractor,  two 
general  categories  can  be  identified:  (1)  those  programs  in  which  the  use 
of  simulation  is  dominant  and  in  which  other  .media  play  a relatively  minor 
role;  and  (2)  those  in  which  simulation  is  used  in  somewhat  more  of  an  ad- 
junctive role  with  other  media  playing  a relatively  greater  role.  These  two 
types  of  programs  differ  not  in  terms  of  the  presence  of  simulation,  but  in 
its  manner  of  use  and  in  the  magnitude  of  the  other  media  requirements.  To 
overstate  this  distinction,  perhaps,  the  one  can  be  characterized  as  simu- 
lator-centered, and  the  other  as  media-centered.  This  dimension  has  con- 
siderable potential  effect  on  the  cost  effectiveness  of  an  ISD  effort.  Both 
categories  of  programs  have  essentially  the  same  simulator  cost  (though  they 
differ  in  the  extent  and  manner  of  its  use),  but  the  costs  associated  with 
the  non-simulator  media  can  differ  significantly.  Heavily  mediated  programs 
are  costly  to  produce,  and  they  are  costly  to  maintain  and  update.  Further, 
it  is  contended  here  that  they  will  contain  content  not  required  in  the 
simulator-centered  program  and  that  they  can  differ  significantly  in  terms 
of  training  effectiveness  as  well.  Therefore,  as  the  Navy  seeks  to  develop 
and  improve  its  aviation  ISD  model  further,  this  dimension  warrents  addi- 
tional research  to  provide  proper  orientation  of  the  model.  At  present,  for 
reasons  previously  discussed  concerning  the  limited  extent  to  which  proce- 
dural guidance  on  this  point  exists  or  has  been  articulated,  the  model  is 
relatively  more  media  centered  and  less  simulator  centered  than  would  be 
desired. 
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Finally,  the  question  must  be  raised  concerning  innovation  and  progress 
in  the  area  of  training  technology.  Obviously,  the  point  of  developing 
process  models  and  specifications  of  procedures  and  products  is  to  maximize 
the  likelihood  of  effective  application  of  the  latest  and  most  advantageous 
technology.  At  the  same  time,  models  and  specifications  must  not  constrain 
new  ideas  or  alternative  approaches  to  the  point  that  inhibits  progress  or 
that  results  in  stereotypic  iteration  of  effective,  but  sub-optimal,  proce- 
dures and  approaches.  Instructional  Systems  Development,  as  a technology, 
has  not  reached  the  point  at  which  it  should  be  so  firmly  cast  that  further 
change  is  not  desired.  The  Navy  ISD  efforts  have  advanced  both  the  tech- 
nology and  its  application  significantly,  and  it  is  recommended  that  the  areas 
identified  in  this  report  be  pursued  as  a means  of  continuing  that  trend. 
Considerable  gains  have  been  made,  but  much  remains  to  be  done. 
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